Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Ephemeral Dataproc Clusters for Spark Workflows
Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise data engineering team is designing a daily data pipeline in Cloud Composer to orchestrate a multi-step analytical workflow. The pipeline must load external files from Cloud Storage into BigQuery, execute an aggregation query, spin up compute resources to process a large Apache Spark batch transformation job, clean up compute resources immediately after completion to minimize costs, and adhere to Google Cloud security best practices for identity and access management.
How should the team design this Cloud Composer orchestration workflow?
Explanations
Author a DAG using specialized Google Cloud operators to orchestrate BigQuery jobs and create an ephemeral Dataproc cluster that runs the Spark job and is deleted afterward, assigning a dedicated user-managed service account with the Composer Worker role and scoped resource permissions.
Author a DAG using specialized Google Cloud operators to orchestrate BigQuery jobs and create an ephemeral Dataproc cluster that runs the Spark job and is deleted afterward, assigning a dedicated user-managed service account with the Composer Worker role and scoped resource permissions.
What Specialized Operators and Dedicated Service Accounts Provide
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. When designing scalable, production-grade workflows, Airflow should act strictly as an orchestrator rather than a compute engine. Specialized Google Cloud operators (such as BigQuery and Dataproc operators) offload execution to managed Google Cloud data services while keeping the Composer environment lightweight and stable.
How It Meets All Requirements
- Orchestration of BigQuery & Spark: The DAG utilizes specialized operators to trigger native asynchronous operations in BigQuery and Dataproc, maintaining precise dependency tracking and task state management.
- Cost Optimization via Ephemeral Compute: The workflow creates a Dataproc cluster on demand, submits the Apache Spark batch processing job, and explicitly deletes the cluster in a downstream task upon completion, eliminating idle infrastructure costs.
- Security Best Practices: The environment runs under a dedicated, user-managed service account assigned the Composer Worker (
roles/composer.worker) role, augmented strictly with least-privilege permissions required for Dataproc, Cloud Storage, and BigQuery.
Key Advantages and Capabilities
- Worker Resource Protection: Offloading Spark computation to Dataproc prevents CPU and memory exhaustion on the underlying Airflow worker nodes.
- Automated Lifecycle Management: Workflow tasks handle the provisioning and teardown of transient clusters automatically.
- Least-Privilege Security: Restricting the environment's service account mitigates risks associated with over-privileged service accounts.
Why This Is Optimal
This architecture leverages the decoupled execution model native to Google Cloud: Cloud Composer handles scheduling and dependency management, while BigQuery and Dataproc scale independently to handle heavy data processing workloads.
Author a DAG using the PythonOperator and BashOperator to run Spark scripts and query executions directly inside the Cloud Composer Airflow worker containers using the default Compute Engine service account.
Author a DAG using the PythonOperator and BashOperator to run Spark scripts and query executions directly inside the Cloud Composer Airflow worker containers using the default Compute Engine service account.
What This Approach Does
This approach uses general-purpose execution operators (PythonOperator and BashOperator) to run both the data manipulation logic and compute-heavy Spark processing directly within the local container runtime of the Airflow worker pods.
Why It Doesn't Meet the Requirements
- Resource Contention & Failure: Cloud Composer worker nodes are provisioned to run the orchestration scheduler and light task dispatching. Executing large Spark workloads directly on Airflow workers quickly causes memory exhaustion, crashing worker pods and causing DAG scheduling failures.
- Security Anti-Pattern: Using the default Compute Engine service account violates least-privilege best practices because it typically possesses broad Editor permissions across the Google Cloud project.
Key Limitations
- Airflow workers lack distributed compute frameworks required to scale Spark transformations across large datasets.
- Running compute on worker nodes prevents horizontal autoscaling from functioning effectively for orchestration tasks.
- Default service accounts expose project infrastructure to unnecessary privilege escalation risks.
This Answer Would Be Correct If
The workload consisted only of trivial, lightweight Python string manipulations or directory checks that require negligible memory and zero external service orchestration.
Additional Context
Heavy processing must always be offloaded to dedicated processing engines like Dataproc, Dataflow, or BigQuery rather than executed on Composer worker nodes.
Author a DAG that triggers Google Cloud Workflows via Cloud Functions, while deploying long-running Spark daemon pods directly into the Cloud Composer system GKE cluster namespaces.
Author a DAG that triggers Google Cloud Workflows via Cloud Functions, while deploying long-running Spark daemon pods directly into the Cloud Composer system GKE cluster namespaces.
What This Approach Does
This option bypasses Airflow's native task management by delegating execution to Cloud Functions and Cloud Workflows, while running heavy third-party Spark daemons inside the internal Google Kubernetes Engine (GKE) cluster managed by Cloud Composer.
Why It Doesn't Meet the Requirements
- System Cluster Interference: Deploying unmanaged workloads directly into the Cloud Composer GKE cluster violates the shared responsibility model and risks interfering with core Airflow system components (such as the scheduler, web server, and database proxy).
- Unnecessary Architectural Complexity: Introducing Cloud Functions and Workflows as intermediary layers to orchestrate BigQuery and Spark duplicates the core scheduling capabilities of Cloud Composer.
- Operational Instability: Running external container workloads directly on Composer's internal cluster can cause resource starvation and disrupt automated Composer environment upgrades.
Key Limitations
- The Composer GKE cluster is designed exclusively to host Airflow control plane components and worker tasks.
- Adding intermediate serverless triggers complicates error handling, logging, and retry logic.
- Custom daemons inside the internal cluster bypass Airflow's built-in concurrency controls.
This Answer Would Be Correct If
The workload was built purely as a microservice-based event-driven workflow without requiring Cloud Composer or Apache Airflow.
Additional Context
Google Cloud documentation explicitly advises against installing or running additional components in the environment's GKE cluster that interfere with Composer services.
Author a DAG that connects to a persistent, 24/7 Dataproc cluster using custom SSH operators and retains the cluster between pipeline runs, while granting all Airflow users the Project Owner IAM role.
Author a DAG that connects to a persistent, 24/7 Dataproc cluster using custom SSH operators and retains the cluster between pipeline runs, while granting all Airflow users the Project Owner IAM role.
What This Approach Does
This configuration keeps a static Dataproc cluster continuously running around the clock and uses generic SSH or remote script execution operators to trigger batch Spark jobs.
Why It Doesn't Meet the Requirements
- Cost Inefficiency: Maintaining a persistent 24/7 Dataproc cluster for a once-daily batch workflow results in substantial idle compute charges, violating the requirement to optimize costs through immediate post-completion cleanup.
- Severe Security Vulnerability: Assigning the Project Owner role to all users violates core IAM governance and the principle of least privilege, providing unrestricted control over all cloud project assets.
- Fragile Orchestration: Executing jobs via raw SSH lacks native monitoring, Airflow lineage integration, and standard retries provided by specialized Google Cloud operators.
Key Limitations
- Static clusters require manual sizing and continuous patching overhead.
- SSH operators do not integrate cleanly with Cloud Logging or Airflow XCom monitoring for Dataproc job states.
- Over-privileged IAM assignments compromise enterprise audit and regulatory compliance.
This Answer Would Be Correct If
The organization had a requirement for sub-second job start times across hundreds of continuous streaming or interactive queries throughout the entire 24-hour cycle.
Additional Context
Cloud Composer provides first-class support for creating and deleting ephemeral Dataproc clusters within the lifecycle of a single DAG run.
Try 10 interactive questions like this one, free
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
- 1,353 practice questions across all 5 exam domains
- Every answer explained: why it is right, and why the others are not
- Scout, an AI tutor that answers your follow-up questions
- Readiness tracking by domain, so you know when to book the exam
Related questions
Keep the momentum going with these hand-picked practice scenarios
AdvancedProfessional Data Engineer
Cloud Composer Highly Resilient Private IP
Job automation and orchestration (e.g., Cloud Composer and Workflows)
AdvancedProfessional Data Engineer
Eventarc to Cloud Workflows
Job automation and orchestration (e.g., Cloud Composer and Workflows)
AdvancedProfessional Data Engineer
Eventarc Asynchronous Event-Driven Architecture
Designing system for fault tolerance and managing restarts
AdvancedProfessional Data Engineer
Pub/Sub Avro Schema Revision Deserialization
Data acquisition and import
AdvancedProfessional Data Engineer
Spanner INSERT_OR_UPDATE Mutation Idempotency
Processing logic
AdvancedProfessional Cloud Security Engineer
Binary Authorization Attestor Policy Rules
Configuring Binary Authorization to secure GKE clusters or Cloud Run.
Last updated Validated