A data engineering team operates a large Apache Spark batch workload on a Google Cloud Dataproc cluster, with input and output datasets stored directly in Cloud Storage buckets. The workload experiences unpredictable processing spikes throughout the day, interspersed with low-utilization periods. During review of recent cluster operations and billing metrics, the team identifies several performance and cost issues: Minor, short-lived fluctuations in resource demand trigger frequent scaling thrashing (unnecessary scale-up and scale-down cycles). Scaling down primary worker instances causes HDFS block movement overhead and delays worker removal. In-progress Spark tasks are abruptly cancelled during node deprovisioning, causing job retries and increased execution costs. Which Dataproc autoscaling policy configuration resolves these issues while minimizing infrastructure costs?

Set minInstances equal to maxInstances in workerConfig, scale elastic capacity using secondaryWorkerConfig, specify a gracefulDecommissionTimeout, and set non-zero values for scaleUpMinWorkerFraction and scaleDownMinWorkerFraction.: What Dataproc Autoscaling Policy Optimization Is Dataproc autoscaling policies allow organizations to automate horizontal scaling of cluster resources based on Hadoop YARN memory and CPU metrics, dynamically adjusting worker nodes to align compute capacity with incoming workload demands while enforcing architectural boundaries. How It Meets the Requirements Preventing Primary Worker Scaling Overhead: Setting minInstances equal to maxInstances within workerConfig keeps the primary worker group at a static, fixed size. Because primary workers run HDFS DataNodes and essential node services, keeping them static prevents expensive HDFS block replication and eliminates the decommission delays associated with data redistribution. Optimizing Cost with Secondary Workers: Configuring elastic scaling bounds in secondaryWorkerConfig ensures that burst compute capacity is fulfilled exclusively by secondary (preemptible or Spot) workers, drastically lowering compute unit costs for dynamic Spark workloads. Preventing Premature Task Eviction: Setting a non-zero gracefulDecommissionTimeout in yarnConfig instructs YARN to allow active Spark executors and map/reduce tasks on decommissioned nodes to complete their in-flight processing before terminating the virtual machine, avoiding failed task retries. Eliminating Autoscaling Thrashing: Setting scaleUpMinWorkerFraction and scaleDownMinWorkerFraction to non-zero fractional values establishes a minimum delta threshold required before an autoscaling event is triggered, preventing cluster adjustments from being initiated by negligible fluctuations in resource queues. Key Advantages Cost minimization: Leverages secondary/Spot VM pricing for all burst processing capacity. Execution stability: Avoids task failure cascades and costly stage re-computations. Operational efficiency: Eliminates HDFS re-balancing delays during cluster contraction. Why This Is Optimal This architecture decouples the persistent cluster infrastructure from stateless, bursty worker execution, fulfilling performance SLAs while maintaining rigid cost governance.