Your enterprise data team is architecting a multi-step data processing pipeline on Google Cloud that orchestrates dependencies between long-running BigQuery batch transformation jobs and interactive analytical workloads. The design must satisfy the following technical and operational requirements: Orchestrate complex task dependencies, retries, and data transformations across hybrid services using Python. Prevent worker resource starvation and task timeouts while waiting for long-running BigQuery batch jobs to complete. Ensure end-to-end idempotency so that automated task retries or scheduled pipeline re-executions do not result in duplicate records or inconsistent states. Automatically capture and publish end-to-end data lineage to Dataplex Universal Catalog without requiring custom reporting code. Which orchestration and pipeline architecture should you implement?

Deploy a Cloud Composer environment with Airflow Triggerers enabled; define a DAG using deferrable operators like BigQueryInsertJobOperator (deferrable=True) to execute MERGE statements against partitioned BigQuery tables, and enable Dataplex data lineage integration.: ### What Cloud Composer with Deferrable Operators Is **Cloud Composer** is a fully managed workflow orchestration service built on **Apache Airflow**. By incorporating **Airflow Triggerers** and **deferrable operators**, Cloud Composer enables tasks to yield their execution worker slot and suspend execution while waiting for external asynchronous operations (such as long-running BigQuery batch query jobs) to finish. The lightweight triggerer process monitors the external job status using asynchronous Python event loops, freeing worker capacity for other tasks. ### How It Meets the Requirements - **Orchestrating Complex Dependencies in Python**: Cloud Composer utilizes declarative **Directed Acyclic Graphs (DAGs)** defined entirely in Python, allowing rich dependency structures, dynamic task generation, and granular retry policies. - **Preventing Worker Starvation**: Configuring `deferrable=True` on operators like `BigQueryInsertJobOperator` ensures that the Airflow worker thread is released back to the worker pool while BigQuery executes the batch query asynchronously. - **Ensuring Idempotency**: Utilizing SQL `MERGE` statements targeted at partitioned BigQuery tables guarantees deterministic upsert logic, ensuring identical outputs regardless of task retry counts. - **Automated Data Lineage**: Cloud Composer natively integrates with **Dataplex Universal Catalog** via the `apache-airflow-providers-openlineage` provider, automatically tracking inputs and outputs of `BigQueryInsertJobOperator` without custom code. ### Key Advantages and Capabilities - **Optimized Resource Utilization**: Eliminates idle worker threads blocking on long-running BigQuery queries. - **Out-of-the-Box Lineage**: Automatically registers table-level and column-level lineage within Dataplex. - **Declarative Scalability**: Scales Airflow workers dynamically according to queued workload volume. - **Robust Error Recovery**: Native Airflow retry policies handle transient API and infrastructure failures cleanly. ### Why This Is Optimal This architecture pairs the asynchronous scalability of **Airflow deferrable operators** with idempotent `MERGE` query patterns and automated Dataplex governance, fulfilling all scaling, resilience, and operational observability criteria.

Orchestrate Hybrid Batch Interactive Cloud Composer | professional-cloud-data-engineer

Orchestrate Hybrid Batch Interactive Cloud Composer

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

Your enterprise data team is architecting a multi-step data processing pipeline on Google Cloud that orchestrates dependencies between long-running BigQuery batch transformation jobs and interactive analytical workloads. The design must satisfy the following technical and operational requirements:

Orchestrate complex task dependencies, retries, and data transformations across hybrid services using Python.
Prevent worker resource starvation and task timeouts while waiting for long-running BigQuery batch jobs to complete.
Ensure end-to-end idempotency so that automated task retries or scheduled pipeline re-executions do not result in duplicate records or inconsistent states.
Automatically capture and publish end-to-end data lineage to Dataplex Universal Catalog without requiring custom reporting code.

Which orchestration and pipeline architecture should you implement?

Implement Google Cloud Workflows with standard HTTP connectors to trigger synchronous BigQuery query jobs with INSERT statements, and invoke the Dataplex Lineage REST API manually upon task completion.

Implement Cloud Scheduler to invoke a Cloud Dataflow batch pipeline that reads unpartitioned staging tables and uses TRUNCATE write dispositions to overwrite target BigQuery tables on each scheduled run.

Deploy a Cloud Composer environment with Airflow Triggerers enabled; define a DAG using deferrable operators like BigQueryInsertJobOperator (deferrable=True) to execute MERGE statements against partitioned BigQuery tables, and enable Dataplex data lineage integration.

Deploy a Cloud Composer DAG using standard synchronous PythonOperator tasks that execute continuous time.sleep polling loops against BigQuery batch jobs, appending records into unpartitioned staging tables.

Professional Cloud Data Engineer

Orchestrate Hybrid Batch Interactive Cloud Composer

Related questions