Dataproc Ephemeral Vs Persistent Cluster Optimization

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise data engineering team runs a batch PySpark data processing workload once every night. The workload exhibits highly variable processing duration (between 45 and 90 minutes) and is fault-tolerant, allowing individual task retries without pipeline failure.

The team wants to minimize overall cloud spending by eliminating idle infrastructure costs during off-peak hours while keeping compute resource expenses as low as possible during job execution.

Which Dataproc cluster lifecycle and compute architecture should the team implement?

Deploy an ephemeral Dataproc cluster configured with zero primary worker nodes and 100% secondary preemptible worker nodes.

Maintain a 24/7 persistent Dataproc cluster sized for peak load using 100% standard on-demand Compute Engine worker nodes.

Maintain a persistent Dataproc cluster that uses a cron-based scaling schedule to reduce the worker node count to zero during off-peak daytime hours.

Create an ephemeral, job-scoped Dataproc cluster for each scheduled run, using a minimal primary worker pool combined with secondary (Spot/preemptible) worker nodes configured via an autoscaling policy.

Professional Cloud Data Engineer

Dataproc Ephemeral Vs Persistent Cluster Optimization

Related questions