Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
Your enterprise data team is architecting a multi-step data processing pipeline on Google Cloud that orchestrates dependencies between long-running BigQuery batch transformation jobs and interactive analytical workloads. The design must satisfy the following technical and operational requirements:
Which orchestration and pipeline architecture should you implement?
Google Cloud Workflows is a fully managed, serverless orchestration platform that links HTTP-based services, Google Cloud APIs, and Cloud Functions using YAML or JSON workflow definitions. It is optimized for low-latency, event-driven microservice workflows.
INSERT INTO queries causes duplicate records whenever a step fails and is retried by the workflow engine.The workload consisted of lightweight, sub-second microservices and serverless infrastructure automation tasks rather than complex batch/interactive data pipeline orchestration.
Workflows is designed for chaining lightweight HTTP endpoints and microservices, whereas Cloud Composer (Managed Airflow) is purpose-built for data-driven ETL/ELT pipelines.
This approach uses standard Airflow PythonOperator tasks running synchronous Python code with time.sleep() loops to continually query the BigQuery API until a submitted job transitions from RUNNING to DONE.
PythonOperator does not automatically parse and emit OpenLineage events for BigQuery jobs unless custom lineage inlets and outlets are manually configured in the code.The pipeline ran very short jobs (a few seconds) where spinning up asynchronous triggerers or specialized operators introduced unnecessary overhead.
Airflow deferrable operators were introduced specifically to eliminate the compute and slot inefficiencies caused by blocking sensor loops and time.sleep polling patterns.
Cloud Scheduler is a fully managed enterprise cron job service used to trigger batch tasks or pipelines on a recurring time schedule, such as initiating Cloud Dataflow batch jobs via templates.
WRITE_TRUNCATE across entire unpartitioned target tables destroys historical data during pipeline execution and fails to support granular incremental updates or hybrid interactive workflows.The workload were a standalone, recurring single-step Dataflow batch job reading files from Cloud Storage on a fixed timer rather than a complex multi-step hybrid workflow.
While Dataflow excels at scalable parallel data transformations, complex cross-service dependencies and interactive job sequencing require an orchestration layer like Cloud Composer.
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. By incorporating Airflow Triggerers and deferrable operators, Cloud Composer enables tasks to yield their execution worker slot and suspend execution while waiting for external asynchronous operations (such as long-running BigQuery batch query jobs) to finish. The lightweight triggerer process monitors the external job status using asynchronous Python event loops, freeing worker capacity for other tasks.
deferrable=True on operators like BigQueryInsertJobOperator ensures that the Airflow worker thread is released back to the worker pool while BigQuery executes the batch query asynchronously.MERGE statements targeted at partitioned BigQuery tables guarantees deterministic upsert logic, ensuring identical outputs regardless of task retry counts.apache-airflow-providers-openlineage provider, automatically tracking inputs and outputs of BigQueryInsertJobOperator without custom code.This architecture pairs the asynchronous scalability of Airflow deferrable operators with idempotent MERGE query patterns and automated Dataplex governance, fulfilling all scaling, resilience, and operational observability criteria.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated