A data engineering team runs business-critical Apache Spark batch processing workloads on a Google Cloud Dataproc cluster. The team observes two main operational issues: During sudden spikes in backlogged tasks, the cluster takes too long to scale out compute capacity, causing jobs to risk violating their processing Service Level Agreements (SLAs). After intermediate caching stages complete, the cluster remains at maximum size and fails to scale down idle compute workers. Which configuration should the data engineering team implement to resolve both issues?

Set the Dataproc autoscaling policy scaleUpFactor to 1.0, maintain Spark dynamic allocation enabled, and configure spark.dynamicAllocation.cachedExecutorIdleTimeout.: What Spark Dynamic Allocation and Dataproc Autoscaling Do Dataproc autoscaling pairs with Spark Dynamic Allocation to dynamically adjust cluster compute capacity based on workload demands. In Apache Spark, dynamic allocation requests YARN containers to host Spark executors when tasks backlog, starting with a baseline executor pool and doubling container allocations while unassigned tasks remain. Setting a responsive autoscaling policy allows Dataproc to translate these pending YARN resource requests into physical Compute Engine worker nodes immediately. How It Meets the Requirements Rapid scale-out for strict SLAs: Because Spark dynamic allocation smooths out pending memory spikes by exponentially requesting executors rather than requesting all containers up front, setting scaleUpFactor to 1.0 (100%) instructs Dataproc to aggressively provision nodes for 100% of pending YARN memory requests on every evaluation cycle. Automated scale-down of cached data: By default, Spark retains executors holding cached datasets indefinitely, preventing YARN and Dataproc from decommissioning idle worker nodes. Setting spark.dynamicAllocation.cachedExecutorIdleTimeout allows Spark to safely release idle executors even if they contain cached partitions, enabling Dataproc autoscaling to scale down nodes when resources become available. Key Advantages Aggressive task processing: Maximizes cluster expansion speed during backlog bursts to guarantee processing SLAs. Prevents resource hoarding: Automatically recovers compute resources from completed data transformation stages. Seamless integration: Leverages Dataproc's native integration between YARN resource scheduling and Spark executor allocation. Why This Is Optimal Unlike MapReduce jobs—which submit all map container requests simultaneously and require low scale-up factors (5–10%) to prevent overprovisioning—Spark manages task scheduling incrementally. A high scaleUpFactor of 1.0 combined with cached executor idle timeouts ensures maximum elasticity in both directions.