Dataproc Ephemeral Vs Persistent Cluster Optimization

What Ephemeral Dataproc Clusters with Secondary Workers Are

Ephemeral (job-scoped) Dataproc clusters are created on demand specifically to execute a designated batch workload or workflow template and are automatically deleted immediately upon job completion. Combining ephemeral lifecycles with secondary workers (preemptible VMs or Spot VMs) enables data pipelines to dynamically provision low-cost compute resources for data processing without incurring long-term maintenance overhead.

How It Meets the Requirements

• Eliminate idle infrastructure costs: Because the cluster is created at job trigger time and dismantled right after the PySpark job finishes, the organization incurs zero Compute Engine and Dataproc licensing charges during the daytime off-peak hours.
• Minimize compute resource expenses during execution: Secondary worker nodes run as Spot or preemptible instances, which offer substantial discounts (up to 60–91%) compared to standard on-demand virtual machine pricing.
• Tolerate variable duration and retries: Spark natively handles worker preemption by rescheduling failed map/reduce tasks on remaining nodes. Configuring an autoscaling policy allows the cluster to scale secondary workers up or down depending on YARN memory and vCPU demand throughout the 45-to-90-minute run.

Key Advantages and Capabilities

• Zero idle waste: Ensures compute costs are strictly proportional to the actual execution time of the batch pipeline.
• Workload isolation: Eliminates multi-tenant dependency conflicts, job interference, or residual file corruption between distinct batch runs.
• Optimized master/worker split: Primary standard workers host YARN NodeManagers and preserve stable coordination, while secondary nodes provide burstable, cost-effective worker capacity.
• Automated orchestration: Easily integrated into managed workflows like Cloud Composer, Workflows, or Dataproc Workflow Templates.

Why This Is Optimal

For nightly batch jobs that do not require sub-second startup responsiveness, the 90-second cluster provisioning overhead is negligible compared to the substantial cost savings achieved by eliminating 22+ hours of idle cluster uptime and running processing tasks on discounted Spot/preemptible instances.