A data engineering team runs business-critical Apache Spark batch processing workloads on a Google Cloud Dataproc cluster. The team observes two main operational issues: During sudden spikes in backlogged tasks, the cluster takes too long to scale out compute capacity, causing jobs to risk violating their processing Service Level Agreements (SLAs). After intermediate caching stages complete, the cluster remains at maximum size and fails to scale down idle compute workers. Which configuration should the data engineering team implement to resolve both issues?

Set the Dataproc autoscaling policy scaleUpFactor to 1.0, maintain Spark dynamic allocation enabled, and configure spark.dynamicAllocation.cachedExecutorIdleTimeout.: ### What Spark Dynamic Allocation and Dataproc Autoscaling Do **Dataproc autoscaling** pairs with **Spark Dynamic Allocation** to dynamically adjust cluster compute capacity based on workload demands. In Apache Spark, dynamic allocation requests YARN containers to host Spark executors when tasks backlog, starting with a baseline executor pool and doubling container allocations while unassigned tasks remain. Setting a responsive autoscaling policy allows Dataproc to translate these pending YARN resource requests into physical Compute Engine worker nodes immediately. ### How It Meets the Requirements * **Rapid scale-out for strict SLAs**: Because Spark dynamic allocation smooths out pending memory spikes by exponentially requesting executors rather than requesting all containers up front, setting **`scaleUpFactor` to 1.0 (100%)** instructs Dataproc to aggressively provision nodes for 100% of pending YARN memory requests on every evaluation cycle. * **Automated scale-down of cached data**: By default, Spark retains executors holding cached datasets indefinitely, preventing YARN and Dataproc from decommissioning idle worker nodes. Setting **`spark.dynamicAllocation.cachedExecutorIdleTimeout`** allows Spark to safely release idle executors even if they contain cached partitions, enabling Dataproc autoscaling to scale down nodes when resources become available. ### Key Advantages * **Aggressive task processing**: Maximizes cluster expansion speed during backlog bursts to guarantee processing SLAs. * **Prevents resource hoarding**: Automatically recovers compute resources from completed data transformation stages. * **Seamless integration**: Leverages Dataproc's native integration between YARN resource scheduling and Spark executor allocation. ### Why This Is Optimal Unlike MapReduce jobs—which submit all map container requests simultaneously and require low scale-up factors (5–10%) to prevent overprovisioning—Spark manages task scheduling incrementally. A high `scaleUpFactor` of 1.0 combined with cached executor idle timeouts ensures maximum elasticity in both directions.

Configure Dataproc Autoscaling Spark Workloads | professional-cloud-data-engineer

Configure Dataproc Autoscaling Spark Workloads

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A data engineering team runs business-critical Apache Spark batch processing workloads on a Google Cloud Dataproc cluster. The team observes two main operational issues:

During sudden spikes in backlogged tasks, the cluster takes too long to scale out compute capacity, causing jobs to risk violating their processing Service Level Agreements (SLAs).
After intermediate caching stages complete, the cluster remains at maximum size and fails to scale down idle compute workers.

Which configuration should the data engineering team implement to resolve both issues?

Set the Dataproc autoscaling policy scaleUpFactor to 0.05, disable Spark dynamic allocation, and configure yarn.nodemanager.resource.memory-mb to reserve 50% of node memory for the page cache.

Set the Dataproc autoscaling policy scaleUpFactor to 1.0, maintain Spark dynamic allocation enabled, and configure spark.dynamicAllocation.cachedExecutorIdleTimeout.

Set the Dataproc autoscaling policy scaleUpFactor to 1.0, enable spark.streaming.dynamicAllocation.enabled, and configure spark.executor.instances to a static value.

Set the Dataproc autoscaling policy scaleUpFactor to 0.1, keep default executor idle timeouts, and increase mapreduce.job.reduce.slowstart.completedmaps to 1.0.

Professional Cloud Data Engineer

Configure Dataproc Autoscaling Spark Workloads

Related questions