Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team operates a large Apache Spark batch workload on a Google Cloud Dataproc cluster, with input and output datasets stored directly in Cloud Storage buckets. The workload experiences unpredictable processing spikes throughout the day, interspersed with low-utilization periods.
During review of recent cluster operations and billing metrics, the team identifies several performance and cost issues:
Which Dataproc autoscaling policy configuration resolves these issues while minimizing infrastructure costs?
This option fixes the secondary worker group at a static instance count by matching its minimum and maximum bounds, while attempting to execute autoscaling on the primary worker pool with cooldown constraints removed.
minInstances equals maxInstances in secondaryWorkerConfig, the secondary worker group cannot scale, completely preventing the cluster from provisioning cheap Spot/preemptible nodes for demand bursts.cooldownPeriod: 0s) causes hyper-sensitive oscillation, triggering continuous cluster resize operations.gracefulDecommissionTimeout causes abrupt worker terminations, killing in-flight Spark executors and triggering cascading task retries.The cluster had a constant, non-variable baseline load requiring fixed secondary Spot capacity and ran short stateless tasks where VM termination had zero impact on job state.
Dataproc autoscaling policies allow organizations to automate horizontal scaling of cluster resources based on Hadoop YARN memory and CPU metrics, dynamically adjusting worker nodes to align compute capacity with incoming workload demands while enforcing architectural boundaries.
minInstances equal to maxInstances within workerConfig keeps the primary worker group at a static, fixed size. Because primary workers run HDFS DataNodes and essential node services, keeping them static prevents expensive HDFS block replication and eliminates the decommission delays associated with data redistribution.secondaryWorkerConfig ensures that burst compute capacity is fulfilled exclusively by secondary (preemptible or Spot) workers, drastically lowering compute unit costs for dynamic Spark workloads.gracefulDecommissionTimeout in yarnConfig instructs YARN to allow active Spark executors and map/reduce tasks on decommissioned nodes to complete their in-flight processing before terminating the virtual machine, avoiding failed task retries.scaleUpMinWorkerFraction and scaleDownMinWorkerFraction to non-zero fractional values establishes a minimum delta threshold required before an autoscaling event is triggered, preventing cluster adjustments from being initiated by negligible fluctuations in resource queues.This architecture decouples the persistent cluster infrastructure from stateless, bursty worker execution, fulfilling performance SLAs while maintaining rigid cost governance.
This configuration attempts to eliminate baseline primary worker overhead by setting primary minInstances to zero, routing all work exclusively away from secondary workers using weight overrides and YARN node labels, and applying immediate decommissioning.
dataproc:am.primary_only property due to upstream metric reporting bugs (YARN-9088), which cause YARN to report invalid cluster utilization metrics and break autoscaling decisions.gracefulDecommissionTimeout to zero immediately shuts down worker VMs without waiting for running containers to finish, terminating in-flight Spark tasks and causing failed stages and job retries.Dataproc natively supported YARN node labels for autoscaling and the application code handled external checkpoint restarts without requiring graceful container drainage.
This configuration applies an aggressive scaling policy to primary workers with zero threshold barrier (scaleUpMinWorkerFraction: 0.0), a full scale-down factor (scaleDownFactor: 1.0), and disables Spark dynamic allocation.
scaleUpMinWorkerFraction to 0.0 means the autoscaler will scale the cluster on any fractional worker delta, exacerbating flapping and causing constant VM provisioning and deletion cycles on small workload shifts.The workload required instant single-node scale-out triggers and ran on a zero-scale ephemeral architecture without HDFS or shared batch stages.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated