An enterprise is designing a scalable daily batch processing system to process multi-terabyte log datasets stored in a Cloud Storage data lake. The system must meet the following operational requirements: Decouple data storage from compute resources to reduce idle infrastructure costs. Execute complex Apache Spark transformations at scale without managing long-running VM clusters. Orchestrate the complete end-to-end workflow (Spark processing and loading final aggregated results into partitioned BigQuery tables) with support for automated retries and workflow tracking. Which architecture and orchestration strategy should the enterprise implement?

Use Cloud Composer to orchestrate the pipeline, triggering Dataproc Serverless for Spark batch workloads to process data in Cloud Storage, and loading the resulting aggregates into partitioned BigQuery tables.: ### What Dataproc Serverless and Cloud Composer are **Cloud Composer** is a fully managed workflow orchestration service built on **Apache Airflow** that enables you to author, schedule, and monitor complex data pipelines across Google Cloud services. **Dataproc Serverless** runs batch Apache Spark workloads without requiring manual cluster provisioning, configuration, or continuous infrastructure management. ### How it meets all requirements - **Decouple storage from compute**: Storing raw data in **Cloud Storage** while executing computation on dynamic **Dataproc Serverless** batches ensures compute resources are only allocated during active job execution. - **Execute Spark transformations without managing clusters**: Dataproc Serverless automatically provisions the exact compute capacity required for the Spark batch job and deallocates resources immediately upon completion. - **Orchestrate end-to-end workflow with retries**: Cloud Composer DAGs use native Airflow operators (such as `DataprocCreateBatchOperator` and BigQuery load operators) to sequence tasks, track execution status, manage dependencies, and handle automatic retries. - **Load aggregates into partitioned BigQuery tables**: BigQuery receives the final processed output and organizes it into **partitioned tables**, optimizing downstream query performance and storage costs. ### Key advantages and capabilities - **Zero cluster management**: Eliminates the operational overhead of cluster sizing, node provisioning, and idle VM maintenance. - **Native fault tolerance**: Cloud Composer provides built-in task retries, error alerting, and execution history logging. - **Cost efficiency**: You pay strictly for the compute capacity consumed during the batch execution and query runtime. - **Optimized data layout**: Partitioning BigQuery tables by date or ingestion time limits query scan costs and accelerates analytics. ### Why this is optimal This approach directly separates the storage layer (**Cloud Storage** and **BigQuery**) from the compute layer (**Dataproc Serverless**), providing elasticity, automated scaling, robust pipeline orchestration, and cost optimization.

Scalable Batch Processing Dataproc Serverless Composer Bigquery | professional-cloud-data-engineer

Scalable Batch Processing Dataproc Serverless Composer Bigquery

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise is designing a scalable daily batch processing system to process multi-terabyte log datasets stored in a Cloud Storage data lake.

The system must meet the following operational requirements:

Decouple data storage from compute resources to reduce idle infrastructure costs.
Execute complex Apache Spark transformations at scale without managing long-running VM clusters.
Orchestrate the complete end-to-end workflow (Spark processing and loading final aggregated results into partitioned BigQuery tables) with support for automated retries and workflow tracking.

Which architecture and orchestration strategy should the enterprise implement?

Maintain a persistent Dataproc cluster running HDFS storage, and use Cloud Scheduler to execute batch scripts via SSH on the master node.

Use Workflows to provision custom Compute Engine VMs with local SSDs for each batch run, writing outputs back to Cloud Storage for BigQuery federated queries.

Use Cloud Composer to orchestrate the pipeline, triggering Dataproc Serverless for Spark batch workloads to process data in Cloud Storage, and loading the resulting aggregates into partitioned BigQuery tables.

Provision a single-node Dataproc cluster and use Cloud Tasks to load raw uncompressed data directly into non-partitioned BigQuery tables with search indexes.

Professional Cloud Data Engineer

Scalable Batch Processing Dataproc Serverless Composer Bigquery

Related questions