Configure Dataproc Autoscaling Spark Workloads

What Spark Dynamic Allocation and Dataproc Autoscaling Do

Dataproc autoscaling pairs with Spark Dynamic Allocation to dynamically adjust cluster compute capacity based on workload demands. In Apache Spark, dynamic allocation requests YARN containers to host Spark executors when tasks backlog, starting with a baseline executor pool and doubling container allocations while unassigned tasks remain. Setting a responsive autoscaling policy allows Dataproc to translate these pending YARN resource requests into physical Compute Engine worker nodes immediately.

How It Meets the Requirements

• Rapid scale-out for strict SLAs: Because Spark dynamic allocation smooths out pending memory spikes by exponentially requesting executors rather than requesting all containers up front, setting scaleUpFactor to 1.0 (100%) instructs Dataproc to aggressively provision nodes for 100% of pending YARN memory requests on every evaluation cycle.
• Automated scale-down of cached data: By default, Spark retains executors holding cached datasets indefinitely, preventing YARN and Dataproc from decommissioning idle worker nodes. Setting spark.dynamicAllocation.cachedExecutorIdleTimeout allows Spark to safely release idle executors even if they contain cached partitions, enabling Dataproc autoscaling to scale down nodes when resources become available.

Key Advantages

• Aggressive task processing: Maximizes cluster expansion speed during backlog bursts to guarantee processing SLAs.
• Prevents resource hoarding: Automatically recovers compute resources from completed data transformation stages.
• Seamless integration: Leverages Dataproc's native integration between YARN resource scheduling and Spark executor allocation.

Why This Is Optimal

Unlike MapReduce jobs—which submit all map container requests simultaneously and require low scale-up factors (5–10%) to prevent overprovisioning—Spark manages task scheduling incrementally. A high scaleUpFactor of 1.0 combined with cached executor idle timeouts ensures maximum elasticity in both directions.