Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise data engineering team runs a batch PySpark data processing workload once every night. The workload exhibits highly variable processing duration (between 45 and 90 minutes) and is fault-tolerant, allowing individual task retries without pipeline failure.
The team wants to minimize overall cloud spending by eliminating idle infrastructure costs during off-peak hours while keeping compute resource expenses as low as possible during job execution.
Which Dataproc cluster lifecycle and compute architecture should the team implement?
In Google Cloud Dataproc, secondary workers are non-data nodes designed strictly for compute processing. Dataproc architecture requires a non-zero minimum of primary worker nodes to maintain cluster stability and core Hadoop/Spark services.
Dataproc supported fully workerless compute topologies where the control plane executed all containerized executors externally, which is not how standard Dataproc clusters function.
Best practice dictates provisioning a small base of primary workers (e.g., 2 nodes) to maintain stability, while allowing secondary preemptible nodes to scale out dynamically up to the desired compute ceiling.
Scheduled scaling automatically adjusts the number of primary or secondary worker nodes in a persistent Dataproc cluster based on predictable time schedules (such as cron expressions).
The cluster hosted a centralized metastore service or interactive Web UI tools that team members needed to access throughout the day even when batch execution was dormant.
True ephemeral clusters created via workflow templates completely tear down master instances and associated root persistent disks, avoiding 100% of non-execution compute and storage billing.
Ephemeral (job-scoped) Dataproc clusters are created on demand specifically to execute a designated batch workload or workflow template and are automatically deleted immediately upon job completion. Combining ephemeral lifecycles with secondary workers (preemptible VMs or Spot VMs) enables data pipelines to dynamically provision low-cost compute resources for data processing without incurring long-term maintenance overhead.
For nightly batch jobs that do not require sub-second startup responsiveness, the 90-second cluster provisioning overhead is negligible compared to the substantial cost savings achieved by eliminating 22+ hours of idle cluster uptime and running processing tasks on discounted Spot/preemptible instances.
A persistent Dataproc cluster remains active continuously across days, months, or years, hosting long-running services, interactive notebooks, or shared multi-tenant query engines.
The organization required near-zero startup latency to serve hundreds of continuous ad-hoc interactive SQL queries or real-time streaming pipelines throughout the business day.
Persistent clusters are typically appropriate for shared analytics environments (e.g., Hive metastore access or Jupyter interactive notebooks), but are financially inefficient for isolated nightly batch ETL workloads.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated