Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
Orchestrating Serverless Spark Batches with Cloud Composer
Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise is designing a scalable daily batch processing system to process multi-terabyte log datasets stored in a Cloud Storage data lake.
The system must meet the following operational requirements:
- Decouple data storage from compute resources to reduce idle infrastructure costs.
- Execute complex Apache Spark transformations at scale without managing long-running VM clusters.
- Orchestrate the complete end-to-end workflow (Spark processing and loading final aggregated results into partitioned BigQuery tables) with support for automated retries and workflow tracking.
Which architecture and orchestration strategy should the enterprise implement?
Explanations
Maintain a persistent Dataproc cluster running HDFS storage, and use Cloud Scheduler to execute batch scripts via SSH on the master node.
Maintain a persistent Dataproc cluster running HDFS storage, and use Cloud Scheduler to execute batch scripts via SSH on the master node.
What a persistent Dataproc cluster with HDFS does
This architecture keeps a dedicated cluster of Compute Engine virtual machines running continuously with data stored locally in the Hadoop Distributed File System (HDFS). Workloads are scheduled using Cloud Scheduler triggering remote shell or SSH commands directly against the cluster master node.
Why it doesn't meet the requirement
- Decouple storage from compute: Storing data on persistent HDFS tightly couples storage and compute, forcing the enterprise to pay for idle VM CPUs and attached persistent disks 24/7.
- Avoid managing long-running clusters: Requires continuous operational maintenance, OS patching, software upgrades, and manual scaling.
- Orchestrate complex pipelines: Cloud Scheduler only triggers cron-based webhooks or Pub/Sub messages; it lacks native DAG dependency management, intermediate state tracking, and sophisticated error handling.
Key limitations
- High idle infrastructure costs: Persistent worker nodes consume compute billing even when no batch jobs are running.
- Single point of failure: Relying on custom SSH scripts lacks robust monitoring, retry policies, and granular task auditing.
- Complex scaling: Scaling HDFS storage requires rebalancing cluster nodes and disks.
This answer would be correct if
The workload required sub-second HDFS access latency for legacy applications with continuous 24/7 processing where cluster provisioning overhead cannot be tolerated.
Use Cloud Composer to orchestrate the pipeline, triggering Dataproc Serverless for Spark batch workloads to process data in Cloud Storage, and loading the resulting aggregates into partitioned BigQuery tables.
Use Cloud Composer to orchestrate the pipeline, triggering Dataproc Serverless for Spark batch workloads to process data in Cloud Storage, and loading the resulting aggregates into partitioned BigQuery tables.
What Dataproc Serverless and Cloud Composer are
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow that enables you to author, schedule, and monitor complex data pipelines across Google Cloud services. Dataproc Serverless runs batch Apache Spark workloads without requiring manual cluster provisioning, configuration, or continuous infrastructure management.
How it meets all requirements
- Decouple storage from compute: Storing raw data in Cloud Storage while executing computation on dynamic Dataproc Serverless batches ensures compute resources are only allocated during active job execution.
- Execute Spark transformations without managing clusters: Dataproc Serverless automatically provisions the exact compute capacity required for the Spark batch job and deallocates resources immediately upon completion.
- Orchestrate end-to-end workflow with retries: Cloud Composer DAGs use native Airflow operators (such as
DataprocCreateBatchOperatorand BigQuery load operators) to sequence tasks, track execution status, manage dependencies, and handle automatic retries. - Load aggregates into partitioned BigQuery tables: BigQuery receives the final processed output and organizes it into partitioned tables, optimizing downstream query performance and storage costs.
Key advantages and capabilities
- Zero cluster management: Eliminates the operational overhead of cluster sizing, node provisioning, and idle VM maintenance.
- Native fault tolerance: Cloud Composer provides built-in task retries, error alerting, and execution history logging.
- Cost efficiency: You pay strictly for the compute capacity consumed during the batch execution and query runtime.
- Optimized data layout: Partitioning BigQuery tables by date or ingestion time limits query scan costs and accelerates analytics.
Why this is optimal
This approach directly separates the storage layer (Cloud Storage and BigQuery) from the compute layer (Dataproc Serverless), providing elasticity, automated scaling, robust pipeline orchestration, and cost optimization.
Use Workflows to provision custom Compute Engine VMs with local SSDs for each batch run, writing outputs back to Cloud Storage for BigQuery federated queries.
Use Workflows to provision custom Compute Engine VMs with local SSDs for each batch run, writing outputs back to Cloud Storage for BigQuery federated queries.
What Workflows and custom Compute Engine VMs do
Workflows is a serverless orchestration engine best suited for chaining microservices and HTTP-based APIs. This design provisions custom Compute Engine instances for ad hoc data transformation and queries the resulting Cloud Storage files via BigQuery external tables.
Why it doesn't meet the requirement
- Execute Spark transformations at scale: Manually building and orchestrating MapReduce or custom processing pipelines on raw Compute Engine VMs requires substantial custom code and reinvents distributed computing capabilities already provided by Dataproc.
- Optimized analytics performance: Relying on federated queries against Cloud Storage results in slower analytical performance compared to storing native data in partitioned BigQuery tables.
- Operational efficiency: Managing VM lifecycle, custom image maintenance, and instance spin-up scripts increases operational complexity.
Key limitations
- High custom development overhead: Developers must manage distributed worker coordination, data partitioning, and failure recovery manually.
- Degraded query performance: BigQuery federated queries do not benefit from native Capacitor columnar caching, column statistics, or BigQuery slot optimizations.
This answer would be correct if
The workload consisted of lightweight, event-driven microservices executing quick administrative HTTP calls rather than heavy distributed data processing.
Additional context
Workflows excels at low-latency microservice orchestration, whereas Cloud Composer is optimized for complex, dependency-rich batch data pipelines.
Provision a single-node Dataproc cluster and use Cloud Tasks to load raw uncompressed data directly into non-partitioned BigQuery tables with search indexes.
Provision a single-node Dataproc cluster and use Cloud Tasks to load raw uncompressed data directly into non-partitioned BigQuery tables with search indexes.
What a single-node Dataproc cluster with Cloud Tasks does
A single-node Dataproc cluster runs master and worker processes on a single VM. Cloud Tasks manages independent, asynchronous task queues to dispatch background work.
Why it doesn't meet the requirement
- Scale for multi-terabyte data: A single-node Dataproc cluster cannot horizontally scale to process multi-terabyte datasets efficiently.
- Cost and performance optimization: Loading unpartitioned datasets into BigQuery causes analytical queries to scan entire tables, significantly increasing query costs and latency.
- Complex pipeline orchestration: Cloud Tasks is designed for individual asynchronous message dispatching, not complex DAG workflow orchestration across multiple data services.
Key limitations
- Compute bottleneck: A single VM lacks the distributed processing power needed for large-scale data lake transformations.
- Inefficient table design: Non-partitioned tables cannot prune data partitions during analytical queries, resulting in full-table scans.
- Search index misapplication: Search indexes optimize point lookups and text searching, not large-scale batch analytical aggregations.
This answer would be correct if
You were building an asynchronous web task queue processing individual lightweight files and indexing unstructured text strings for point search queries.
Try 10 interactive questions like this one, free
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
- 1,353 practice questions across all 5 exam domains
- Every answer explained: why it is right, and why the others are not
- Scout, an AI tutor that answers your follow-up questions
- Readiness tracking by domain, so you know when to book the exam
Related questions
Keep the momentum going with these hand-picked practice scenarios
AdvancedProfessional Data Engineer
Dataflow Enrichment with Bigtable
Processing data
AdvancedProfessional Data Engineer
Cloud Composer Custom Lineage Inlets Outlets
Processing data
AdvancedProfessional Data Engineer
Cloud Composer Worker Concurrency vs Max Workers
Creating directed acyclic graphs (DAGs) for Cloud Composer
AdvancedProfessional Data Engineer
Cloud Composer Environment Snapshots
Running jobs in multiple regions or zones
AdvancedProfessional Data Engineer
BigQuery Denormalization and Clustering
Minimizing costs per required business need for data
AdvancedProfessional Cloud Security Engineer
Privileged Access Manager Just-in-Time Grants
Managing a super administrator account.
Last updated Validated