Prepare and test your skills
Prepare and test your skills
Free sample question, answered and explained below. The Professional Cloud DevOps Engineer practice bank has 1,455 more like it. Try 10 free: instant score, no account.
Your team manages an automated batch analytics pipeline deployed on a Google Cloud Dataproc cluster with autoscaling enabled. During analysis of recent billing alerts, you notice that the cluster consistently scales up during Apache Spark job execution but fails to scale down worker nodes after processing completes, remaining at peak worker capacity and incurring excess compute costs.
Observability metrics in Cloud Monitoring show that YARN Pending resource drops to zero and Available resource is high, but Spark executor containers are not being decommissioned because interim data was persisted into memory using DataFrame cache operations.
Which corrective action should you implement to resolve this autoscaling failure?
The yarn.nodemanager.resource.memory-mb property defines the maximum amount of physical memory that YARN NodeManager can allocate to compute containers on an individual worker instance.
0.8 * total memory) for YARN, leaving 20% for OS page caching and system daemons. Halving YARN memory wastes provisioned hardware.Non-YARN host processes (such as custom monitoring daemons or local operating system utilities) were suffering from out-of-memory kernel kills due to YARN container overconsumption.
In Apache Spark on Google Cloud Dataproc, dynamic allocation allows the Spark engine to dynamically request and release executor containers from YARN based on workload demands. However, by default, Spark protects executors hosting cached or persisted in-memory datasets from eviction to prevent costly data recomputation.
spark.dynamicAllocation.cachedExecutorIdleTimeout sets a maximum duration that an executor holding cached data can sit idle before Spark actively evicts it and releases the underlying YARN container.unpersist() or uncaching datasets when downstream transformations are complete frees cluster memory immediately, removing the block on executor removal.This approach directly remediates the root cause of the scaling constraint without disabling autoscaling or compromising execution efficiency for future workloads.
The scaleUpFactor parameter determines the aggressiveness with which the Dataproc autoscaler provisions new worker nodes when YARN reports Pending resource container requests.
scaleUpFactor controls cluster expansion (scale-up) rather than cluster contraction (scale-down).scaleUpFactor to 1.0 (100%) is recommended for bursty Spark workloads during scale-up, but it has zero impact on reclaiming idle nodes pinned by active executors.Your Spark cluster was scaling up too slowly to handle sudden backlogs of tasks, requiring an immediate 100% scale-up response to pending container requests.
This configuration turns off Spark's built-in dynamic allocation mechanism and fixes the number of Spark executor containers to a static number specified by spark.executor.instances at application startup.
You were running dedicated, standalone Spark jobs with perfectly predictable concurrency where dynamic container negotiation overhead needs to be eliminated entirely.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated