Persistent Versus Job-Based Dataproc Clusters

Implementing orchestration and resource isolation patterns means separating tenants, versions, and compute so one pipeline cannot exhaust or expose another. In this pattern, Dataflow (opens in a new tab) projects provide tenant boundaries, Flex Templates packaged as Docker images provide version boundaries, and externalized services provide resource boundaries. The Dataflow backend acts as the control plane that orchestrates the full ephemeral lifecycle from submission to autoscaling to shutdown.

1.1Project-Level Isolation for Multi-Tenant Pipelines

Project-level isolation launches pipeline jobs in a different project from the project that owns the template registry and storage bucket. Developers can use Classic Templates and Flex Templates to run jobs this way, which is useful when data processing for multiple customers must be isolated into separate projects. A Flex Template is encapsulated within a Docker image that packages all pipeline dependencies aside from the Flex Template spec into one deployment artifact managed by Artifact Registry. Access to that artifact is controlled with Docker image management features, such as granting pull and optionally push permissions on Artifact Registry and using image tags for different versions.

1.2Versioned Isolation for Phased Rollouts

Versioned isolation with Flex Templates allows each launch to specify a different version of a Docker image when the pipeline starts. This capability simplifies phased replacement of batch pipelines or streaming pipelines across multiple projects when templates are updated later. Because the image contains the dependencies, an older project can keep running its pinned tag while a newer project validates the updated tag. The result is controlled coexistence of versions instead of a single shared artifact that forces all tenants to upgrade at once.

1.3Externalized Services for Resource Isolation

Externalized services for resource isolation move heavy shuffle and streaming execution out of worker virtual machines (VMs) and into a dedicated service. Streaming Engine moves streaming pipeline execution out of VM workers, while Dataflow Shuffle moves shuffle operations for batch pipelines out of VM workers. Flexible resource scheduling (FlexRS) further reduces batch cost by combining advanced scheduling, the Dataflow Shuffle service, and a mix of preemptible VM instances and regular VMs.

Streaming Engine is recommended for streaming pipelines and is enabled by default with recent Apache Beam Java and Python software development kits (SDKs). Dataflow Shuffle is recommended for batch pipelines and is enabled by default with the Java SDK and latest Python SDK. In some streaming cases, resource usage can be reduced further with at-least-once processing for use cases that can tolerate duplicates.

1.4Orchestrated Lifecycle of Ephemeral Jobs

The orchestrated lifecycle of an ephemeral Dataflow job defines how resources are acquired, used, and released without a persistent cluster. To run a job, the request is submitted to a region and routed to an available Dataflow backend in one zone within that region. Before assignment, Dataflow verifies quota and permissions, which creates a clean admission boundary between orchestration and execution.

1. Backend assignment: The job moves to JOB_STATE_PENDING after preflight checks complete and a backend is assigned, while FlexRS jobs enter JOB_STATE_QUEUED when backend assignment is delayed to a future time.
2. Worker startup: The assigned backend starts Dataflow workers in the Google Cloud project, choosing the worker zone to avoid cross-zone traffic when Dataflow Shuffle is used.
3. Work distribution: Workers request work from the backend, and the backend splits work into parallelizable chunks called bundles that are distributed among workers.
4. Autoscaling: The backend starts more workers when existing workers cannot handle the work and shuts down workers when more workers exist than needed, if autoscaling is enabled.
5. Termination: Streaming jobs remain in JOB_STATE_RUNNING until cancelled or drained unless a failure occurs, while batch jobs stop automatically when processing completes or an unrecoverable error occurs.

During processing, the Dataflow backend remains the control plane that orchestrates execution while the data plane performs shuffle operations. Jobs use either worker-based shuffle with shuffle data stored on persistent disks or service-based shuffle externalized from worker VMs. The service-based path uses Dataflow Shuffle for batch pipelines and Streaming Engine for streaming pipelines, which significantly improves performance and scalability of shuffling compared to worker-based shuffle.

1.5Network and Identity Boundaries for Isolation

Network and identity boundaries isolate single-job pipelines from the internet, from other perimeters, and from over-privileged credentials. By default, Dataflow workers receive public IP addresses to reach the internet and Google application programming interfaces (APIs), but secure pipelines run workers without public IPs by passing --usePublicIps=false. When public IPs are disabled, workers route traffic to services such as Cloud Storage, BigQuery, or Pub/Sub internally by enabling Private Google Access on the Virtual Private Cloud (VPC) subnet. Stronger perimeters place pipelines inside VPC Service Controls (VPC-SC) so data cannot be copied outside the perimeter even by users with valid credentials, use Shared VPCs with dedicated subnets and strict firewall rules, and explicitly disable Secure Shell (SSH).

Identity isolation separates the submitter from the runtime through least-privilege service accounts (opens in a new tab). Dataflow otherwise attempts to use the Compute Engine default service account, which often has broad Editor permissions, so production pipelines create a dedicated user-managed worker service account passed at submission. The worker service account receives roles/dataflow.worker plus only the exact read or write roles on touched resources, such as roles/storage.objectViewer on an input bucket and roles/bigquery.dataEditor on a target table. The developer or continuous integration and continuous delivery (CI/CD) pipeline that submits the job needs roles/dataflow.admin and roles/iam.serviceAccountUser to act as the worker account, while the worker VMs retain only runtime execution permissions.

Evaluating cost and latency trade-offs in cluster lifecycles means choosing whether capacity is paid for continuously or only during execution. A Dataproc (opens in a new tab) persistent cluster stays running to serve repeated work, while a job-based ephemeral cluster moves through provisioning, running, and deleted states for each job. Persistent clusters trade continuous virtual machine (VM) charges for near-zero startup latency, while ephemeral clusters trade per-job startup overhead for zero idle cost between jobs.

2.1Persistent Versus Ephemeral Cluster Economics

Persistent clusters and ephemeral clusters represent opposite answers to the same cost model question of whether capacity should be paid for continuously or only during execution. A persistent cluster keeps primary workers, control processes, and local state alive across jobs, so interactive queries, streaming pipelines, and back-to-back batch jobs start without provisioning delay but continue to incur compute charges during idle periods. A job-based cluster provisions VMs when a job is scheduled and deletes them when the job finishes, so starting and stopping VMs incurs repeated overhead but no cost accrues when no job is running. Benchmark and profile workloads accordingly, then categorize them by frequency, duration, and latency sensitivity before committing to either lifecycle.

Persistent capacity that carries a stable minimum load can be covered by committed-use discounts (CUDs), while variable peaks remain on ephemeral or autoscaled capacity to avoid overprovisioning.

2.2Startup Overhead and Idle Cost Trade-off

Startup overhead in an ephemeral lifecycle includes VM provisioning, redistribution of work to new workers, and initialization of job stages that execute sequentially. Avoiding very small jobs improves efficiency because fewer jobs on larger datasets amortize this fixed startup cost, and setting the initial number of workers avoids the extra cost of redistributing work during scale-up. Persistent lifecycles avoid that repeated overhead but pay for unused capacity, which is why development, testing, and experimental environments favor ephemeral infrastructure when data persistence is not necessary. For production environments that require high availability, predictable performance, and operational stability, the idle cost of a persistent cluster is the price of eliminating scale-up latency for bursts and spikes.

2.3Autoscaling Policies and Scale-Down Behavior

An autoscaling policy in a persistent cluster dynamically adds workers when load rises and removes them when load falls, so cost follows demand without deleting the cluster itself. For fluctuating loads, autoscaling ensures that no cost is incurred when load is low while maintaining sufficient capacity for current demand, and maximum scaling limits keep spending within budget. Scale-down speed depends on isolation: separating batch work onto dedicated pools with labels, taints, and tolerations lets the autoscaler delete empty nodes faster because it does not need to restart pods, while system processes that cannot be restarted must run on different pools so they do not permanently block scale-down. The optimize-utilization profile further accelerates scale-down, and node auto-provisioning automatically creates dedicated pools for jobs with matching taints without predefining every pool.

Higher utilization targets can lead to lower costs but worse tail latency, especially for varying loads, so latency-sensitive pipelines should test per-worker capacity under real load before raising targets. For streaming-style continuous work, use autoscaling hints such as minimum and maximum worker ranges and worker utilization hints to tune the trade-off between extra capacity for spikes and steady-state cost.

2.4Secondary Workers and Spot Capacity

Spot VMs (opens in a new tab) are Compute Engine VM instances with no availability guarantees that cost up to 91% less than standard VMs, and they function as low-cost secondary capacity for fault-tolerant work. Spot capacity is best suited for batch or fault-tolerant jobs that are less sensitive to ephemeral behavior, while stateful and serving workloads must not use Spot VMs unless the system is explicitly designed to handle preemption. Three constraints govern their use:

• Pod Disruption Budgets might not be respected because Spot VMs can shut down inadvertently.
• There is no guarantee of graceful shutdown once preemption ignores the pod grace period.
• It might take several minutes to detect that the node was preempted, which delays rescheduling work to a new node.

Because the autoscaler is optimized for infrastructure cost and gives preference to Spot VMs, maintain a backup pool without Spot VMs to ensure that a node is always available. Check Spot VM availability before provisioning, and combine preemptible or Spot VMs with standard VMs so the pipeline makes progress even if Compute Engine preempts the discounted capacity. For broadly applicable savings without preemption risk, E2 VMs offer 31% savings compared to N1 machine types for web, microservice, and small-to-medium database workloads.

2.5Region, Locality, and Data Movement Costs

Cluster region selection directly trades compute price against latency and data movement cost. Cost varies per computing region, so run the workload in the least expensive option where latency does not affect the customer, using the Google Cloud Region Picker to compare cost, sustainability, and latency factors. Run the job in the same region as the data stores, messaging systems, and Cloud Storage buckets used for staging and temporary files, because copying data across regions to run a batch job adds both transfer charges and delay. For production topologies that span zones for availability, monitor inter-zone egress traffic and colocate dependent processes in the same zone to minimize both cost and network latency, while non-production ephemeral clusters favor single-zone placement to save egress charges.

2.6Budgets, Quotas, and Continuous Monitoring

Cost control for either lifecycle starts with defined throughput and latency service level objectives (SLOs), because low end-to-end latency, late-arriving data, and extra capacity for spikes all increase cost. Before production, run smaller jobs on a subset of data to estimate cost, monitor estimated spend on the Cost page, export Cloud Billing data to BigQuery for usage analysis, and create Cloud Monitoring alerts when a job exceeds a defined threshold. Enforce financial boundaries with detailed project budgets, quotas at project, service, and resource levels, proactive quota alerts, and organization policies that limit allowed locations and products. If long-term commitment is planned, purchase committed-use discounts for one year or three years at up to 70% discount by committing to the minimum baseline usage, such as nighttime load, and let ephemeral or autoscaled capacity handle the remainder.

Designing architectures for state externalization and metadata persistence means moving data and metadata out of a Dataproc cluster so the cluster can be deleted without losing work. In this architecture, Cloud Storage holds the persistent files and Dataproc Metastore holds the shared schemas, while Dataproc clusters provide only temporary compute for independent, job-based runs. This separation allows each job to run on a right-sized, isolated cluster and still see the same tables and data as the previous run.

3.1Cloud Storage as the Persistent File System

Cloud Storage (opens in a new tab) as the persistent file system is the durable data lake location that survives after a job-based Dataproc cluster is deleted. Data flows from ingestion and processing jobs into Cloud Storage buckets, and later Dataproc clusters read that same data back from Cloud Storage on their next run, so no state depends on the cluster's local disks. Cloud Storage is designed for 11 nines of durability with synchronous replication within a region for regional buckets and asynchronous replication across regions for dual-region or multi-region buckets, which supports recovery from zone failures and ransomware events. Because storage persists independently of compute, development and testing workloads can use ephemeral infrastructure when data persistence on the cluster is not necessary, while production data remains available for the next job.

Persistent data in Cloud Storage still needs cost and recovery planning to stay efficient across many cluster runs. Object Lifecycle Management automatically transitions objects to lower-cost storage classes based on age or access patterns, and Autoclass dynamically moves data to the most cost-effective class based on usage patterns. Recovery planning must align the processing pipeline and the data it depends on to the same recovery time objective, because a pipeline cannot meet its end-to-end recovery target if its source data recovers more slowly. Historical data that has already been processed into another system, such as daily sales records exported to Cloud Storage and then imported to BigQuery, can tolerate a higher recovery point objective than unprocessed data that exists only in the lake.

3.2Dataproc Metastore as the Shared Metadata Layer

Dataproc Metastore as the shared metadata layer is a fully managed, highly available, autohealing, and serverless Apache Hive metastore that provides data abstraction and data discovery across Dataproc clusters. The metastore stores table definitions, schemas, and locations separately from compute, so multiple independent Dataproc clusters can attach to the same Dataproc Metastore service and see consistent tables even though each cluster has its own lifecycle. This shared layer removes the need to rebuild schema knowledge on every cluster and prevents divergence when teams run isolated, job-scoped clusters for cost efficiency and resource isolation.

The Dataproc Metastore service supports externalized resilience operations that keep metadata recoverable outside any single cluster:

• Backup creates a restorable copy of metastore metadata inside the service.
• Restore recovers the metastore from a backup after a disaster or corruption.
• Export writes metadata to an external location for long-term retention.
• Import loads previously exported metadata into a metastore.

An architecture can add an export task alongside the extract, transform, load (ETL) job to maintain an external backup with each run. If the metastore and the files in Cloud Storage drift out of sync, the metastore can be recreated from the data itself by using the MSCK repair command, which rebuilds table partitions from the underlying file layout.

3.3Aligning Persistence Strategy with Workload and Environment

The persistence strategy is the decision about when to use a long-lived cluster and when to use short-lived, job-based clusters backed by Cloud Storage and Dataproc Metastore. Production environments require high availability, predictable performance, operational stability, and security with robust resources, while development and testing environments prioritize cost efficiency, flexible burstable capacity, and ephemeral infrastructure. Choosing environment-specific resources ensures that persistent clusters are reserved for workloads that need continuous availability, and that transient workloads do not pay for idle compute between runs.

Workload characteristics determine how much isolation, scaling, and durability each job needs:

A job-based Dataproc design fits non-critical, event-driven, and experimental patterns because the cluster exists only for the duration of the job and state remains externalized. The cluster depends on Cloud Storage for input and output data and depends on Dataproc Metastore for schema resolution, so deleting the cluster after the job ends reduces cost without breaking the next run. Mission-critical pipelines that need continuous availability may still justify persistent capacity, but even those designs benefit from externalized backups of both data and metadata to meet recovery and isolation goals.

Key Facts

• Persistent Dataproc clusters incur idle resource cost but provide low startup latency, while ephemeral job-based clusters incur per-job startup overhead but incur no cost when no job is running.
• Flexible resource scheduling (FlexRS) reduces batch cost with delayed backend assignment and a mix of preemptible and regular VMs, and FlexRS jobs enter JOB_STATE_QUEUED while standard jobs enter JOB_STATE_PENDING after backend assignment.
• Streaming Engine externalizes streaming execution and Dataflow Shuffle externalizes batch shuffle operations such as GroupByKey from worker VMs to dedicated services.
• Dataproc state externalization stores files in Cloud Storage and schemas in Dataproc Metastore, so independent clusters can be deleted after each run without losing data or metadata.
• Spot VMs cost up to 91% less than standard VMs but provide no availability guarantees and must be paired with standard VMs for reliable progress.