A data engineering team operates a large Apache Spark batch workload on a Google Cloud Dataproc cluster, with input and output datasets stored directly in Cloud Storage buckets. The workload experiences unpredictable processing spikes throughout the day, interspersed with low-utilization periods. During review of recent cluster operations and billing metrics, the team identifies several performance and cost issues: Minor, short-lived fluctuations in resource demand trigger frequent scaling thrashing (unnecessary scale-up and scale-down cycles). Scaling down primary worker instances causes HDFS block movement overhead and delays worker removal. In-progress Spark tasks are abruptly cancelled during node deprovisioning, causing job retries and increased execution costs. Which Dataproc autoscaling policy configuration resolves these issues while minimizing infrastructure costs?

Set minInstances equal to maxInstances in workerConfig, scale elastic capacity using secondaryWorkerConfig, specify a gracefulDecommissionTimeout, and set non-zero values for scaleUpMinWorkerFraction and scaleDownMinWorkerFraction.: ### What Dataproc Autoscaling Policy Optimization Is Dataproc autoscaling policies allow organizations to automate horizontal scaling of cluster resources based on Hadoop YARN memory and CPU metrics, dynamically adjusting worker nodes to align compute capacity with incoming workload demands while enforcing architectural boundaries. ### How It Meets the Requirements - **Preventing Primary Worker Scaling Overhead:** Setting **`minInstances` equal to `maxInstances`** within **`workerConfig`** keeps the primary worker group at a static, fixed size. Because primary workers run HDFS DataNodes and essential node services, keeping them static prevents expensive HDFS block replication and eliminates the decommission delays associated with data redistribution. - **Optimizing Cost with Secondary Workers:** Configuring elastic scaling bounds in **`secondaryWorkerConfig`** ensures that burst compute capacity is fulfilled exclusively by secondary (preemptible or Spot) workers, drastically lowering compute unit costs for dynamic Spark workloads. - **Preventing Premature Task Eviction:** Setting a non-zero **`gracefulDecommissionTimeout`** in `yarnConfig` instructs YARN to allow active Spark executors and map/reduce tasks on decommissioned nodes to complete their in-flight processing before terminating the virtual machine, avoiding failed task retries. - **Eliminating Autoscaling Thrashing:** Setting **`scaleUpMinWorkerFraction`** and **`scaleDownMinWorkerFraction`** to non-zero fractional values establishes a minimum delta threshold required before an autoscaling event is triggered, preventing cluster adjustments from being initiated by negligible fluctuations in resource queues. ### Key Advantages - **Cost minimization:** Leverages secondary/Spot VM pricing for all burst processing capacity. - **Execution stability:** Avoids task failure cascades and costly stage re-computations. - **Operational efficiency:** Eliminates HDFS re-balancing delays during cluster contraction. ### Why This Is Optimal This architecture decouples the persistent cluster infrastructure from stateless, bursty worker execution, fulfilling performance SLAs while maintaining rigid cost governance.

Dataproc Autoscaling Policy Cost Optimization | professional-cloud-data-engineer

Dataproc Autoscaling Policy Cost Optimization

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A data engineering team operates a large Apache Spark batch workload on a Google Cloud Dataproc cluster, with input and output datasets stored directly in Cloud Storage buckets. The workload experiences unpredictable processing spikes throughout the day, interspersed with low-utilization periods.

During review of recent cluster operations and billing metrics, the team identifies several performance and cost issues:

Minor, short-lived fluctuations in resource demand trigger frequent scaling thrashing (unnecessary scale-up and scale-down cycles).
Scaling down primary worker instances causes HDFS block movement overhead and delays worker removal.
In-progress Spark tasks are abruptly cancelled during node deprovisioning, causing job retries and increased execution costs.

Which Dataproc autoscaling policy configuration resolves these issues while minimizing infrastructure costs?

Set minInstances to 0 in workerConfig, set weight to 0 in secondaryWorkerConfig, set gracefulDecommissionTimeout to 0, and configure YARN node labels with dataproc:am.primary_only.

Set minInstances equal to maxInstances in workerConfig, scale elastic capacity using secondaryWorkerConfig, specify a gracefulDecommissionTimeout, and set non-zero values for scaleUpMinWorkerFraction and scaleDownMinWorkerFraction.

Configure primary worker scaling with scaleUpMinWorkerFraction set to 0.0, set scaleDownFactor to 1.0, and configure spark.dynamicAllocation.enabled=false on the cluster.

Set minInstances equal to maxInstances in secondaryWorkerConfig, enable dynamic autoscaling on workerConfig with a 0-second cooldown period, and remove gracefulDecommissionTimeout.

Professional Cloud Data Engineer

Dataproc Autoscaling Policy Cost Optimization

Related questions