An enterprise analytics team manages batch Spark workloads for multiple internal business units. The workloads have the following operational requirements and constraints: Each business unit requires strict resource isolation, distinct package dependencies, and precise cost attribution. The batch jobs run intermittently or once daily on varying schedules rather than continuously. Compute costs must be minimized by eliminating idle cluster resources between scheduled executions. Persistent data must remain decoupled from the compute resources to avoid data loss. Which architecture and orchestration strategy should the team implement?

Author Cloud Composer DAGs that create an ephemeral Dataproc cluster configured with tenant-specific dependencies, execute the batch job, stage and output data to Cloud Storage, and delete the cluster upon job completion.: ### What Ephemeral Dataproc Clusters with Cloud Composer Are **Cloud Composer** acts as an orchestration engine using Apache Airflow Directed Acyclic Graphs (DAGs) to programmatically manage the lifecycle of **ephemeral (job-scoped) Dataproc clusters**. In this pattern, the cluster is dynamically provisioned just before job execution and deleted immediately following completion, while all persistent input and output data resides externally in **Cloud Storage**. ### How It Meets the Requirements - **Strict Resource and Dependency Isolation**: Provisioning dedicated, single-job clusters prevents noisy-neighbor CPU/memory contention and ensures tenant-specific software configurations or custom image versions (`--image-version`) do not conflict across different teams. - **Cost Optimization**: Ephemeral lifecycles eliminate idle compute costs between intermittent or daily runs by tearing down Compute Engine virtual machines as soon as processing finishes. - **Precise Cost Attribution**: Cluster labels and dedicated per-job provisioning enable granular billing tracking per business unit. - **Storage and Compute Separation**: Decoupling compute from data storage using Cloud Storage ensures persistent data remains unaffected by cluster destruction. ### Key Advantages - **No Idle Waste**: Pay only for the compute and storage resources used during active data processing. - **Dependency Stability**: Custom images, initialization scripts, and job uber-JARs are isolated per cluster, preventing runtime collisions. - **Automated Lifecycle Management**: Airflow operators (`DataprocCreateClusterOperator`, `DataprocSubmitJobOperator`, `DataprocDeleteClusterOperator`) fully automate workflow execution. - **Simplified SLA Management**: Dedicated cluster resources guarantee that one tenant's long-running job does not starve another tenant's critical pipeline. ### Why This Is Optimal For intermittent, scheduled batch workloads with heterogeneous dependencies and multi-tenant billing requirements, ephemeral Dataproc clusters orchestrated via Cloud Composer offer the best balance of cost efficiency, fault tolerance, and security boundary isolation.

Orchestrate Ephemeral Dataproc Clusters Composer | professional-cloud-data-engineer

Orchestrate Ephemeral Dataproc Clusters Composer

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise analytics team manages batch Spark workloads for multiple internal business units. The workloads have the following operational requirements and constraints:

Each business unit requires strict resource isolation, distinct package dependencies, and precise cost attribution.
The batch jobs run intermittently or once daily on varying schedules rather than continuously.
Compute costs must be minimized by eliminating idle cluster resources between scheduled executions.
Persistent data must remain decoupled from the compute resources to avoid data loss.

Which architecture and orchestration strategy should the team implement?

Author Cloud Composer DAGs that create an ephemeral Dataproc cluster configured with tenant-specific dependencies, execute the batch job, stage and output data to Cloud Storage, and delete the cluster upon job completion.

Deploy a single, long-running persistent Dataproc cluster shared across all business units, enabling YARN FairScheduler and autoscaling to handle concurrent tenant jobs.

Execute all tenant Spark data transformations directly inside Cloud Composer worker pods using the PythonOperator and local PySpark libraries.

Maintain a single persistent Dataproc cluster and use Cloud Composer DAGs to execute dynamic initialization action scripts against the master node before every job submission.

Professional Cloud Data Engineer

Orchestrate Ephemeral Dataproc Clusters Composer

Related questions