Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team runs business-critical Apache Spark batch processing workloads on a Google Cloud Dataproc cluster. The team observes two main operational issues:
Which configuration should the data engineering team implement to resolve both issues?
A low scaleUpFactor of 0.05 (5%) restricts cluster expansion to only a small fraction of pending YARN memory requests per cycle. Disabling Spark dynamic allocation forces Spark to allocate all executor containers statically at the start of the job, while overriding yarn.nodemanager.resource.memory-mb alters the memory allocation boundary between YARN daemons and the operating system.
scaleUpFactor of 0.05 is intended for MapReduce jobs with massive initial container request spikes. For Spark jobs, this severely throttles node provisioning during task backlogs and directly violates processing SLAs.spark.dynamicAllocation.enabled=false) stops Spark from dynamically expanding or releasing containers as execution stages progress.This approach would be suitable for legacy Hadoop MapReduce workloads characterized by short map tasks with massive, transient container request spikes where gradual cluster expansion prevents overprovisioning.
Dataproc autoscaling pairs with Spark Dynamic Allocation to dynamically adjust cluster compute capacity based on workload demands. In Apache Spark, dynamic allocation requests YARN containers to host Spark executors when tasks backlog, starting with a baseline executor pool and doubling container allocations while unassigned tasks remain. Setting a responsive autoscaling policy allows Dataproc to translate these pending YARN resource requests into physical Compute Engine worker nodes immediately.
scaleUpFactor to 1.0 (100%) instructs Dataproc to aggressively provision nodes for 100% of pending YARN memory requests on every evaluation cycle.spark.dynamicAllocation.cachedExecutorIdleTimeout allows Spark to safely release idle executors even if they contain cached partitions, enabling Dataproc autoscaling to scale down nodes when resources become available.Unlike MapReduce jobs—which submit all map container requests simultaneously and require low scale-up factors (5–10%) to prevent overprovisioning—Spark manages task scheduling incrementally. A high scaleUpFactor of 1.0 combined with cached executor idle timeouts ensures maximum elasticity in both directions.
Enabling spark.streaming.dynamicAllocation.enabled=true activates streaming-specific executor scaling based on micro-batch processing signals. Setting spark.executor.instances statically defines the exact number of executors Spark should run, which directly disables dynamic elasticity.
DStreams) using batch processing time signals, not standard Spark batch workloads or Spark Core dynamic allocation.spark.executor.instances fixes the executor count, preventing Spark Core from requesting additional containers when backlogs grow or shedding containers when idle.spark.dynamicAllocation.cachedExecutorIdleTimeout, meaning executors retaining cached blocks still cannot be safely decommissioned.The workload was a Spark Streaming (DStreams) application where executor scaling needs to be triggered by micro-batch scheduling delays rather than YARN pending task queues.
Setting scaleUpFactor to 0.1 (10%) limits worker scaling increments to 10% of pending container demand. The parameter mapreduce.job.reduce.slowstart.completedmaps is a Hadoop MapReduce setting that dictates what percentage of map tasks must complete before reduce task containers are scheduled.
mapreduce.job.reduce.slowstart.completedmaps has zero effect on Apache Spark execution, as Spark uses its own DAG scheduler (stages and shuffle dependencies) rather than Hadoop MapReduce reducers.The cluster were running native Hadoop MapReduce jobs where reducer scheduling needed to be delayed until 100% of mappers completed to avoid blocking cluster slots.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated