Your enterprise data team is architecting a multi-step data processing pipeline on Google Cloud that orchestrates dependencies between long-running BigQuery batch transformation jobs and interactive analytical workloads. The design must satisfy the following technical and operational requirements: Orchestrate complex task dependencies, retries, and data transformations across hybrid services using Python. Prevent worker resource starvation and task timeouts while waiting for long-running BigQuery batch jobs to complete. Ensure end-to-end idempotency so that automated task retries or scheduled pipeline re-executions do not result in duplicate records or inconsistent states. Automatically capture and publish end-to-end data lineage to Dataplex Universal Catalog without requiring custom reporting code. Which orchestration and pipeline architecture should you implement?

Deploy a Cloud Composer environment with Airflow Triggerers enabled; define a DAG using deferrable operators like BigQueryInsertJobOperator (deferrable=True) to execute MERGE statements against partitioned BigQuery tables, and enable Dataplex data lineage integration.: What Cloud Composer with Deferrable Operators Is Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. By incorporating Airflow Triggerers and deferrable operators, Cloud Composer enables tasks to yield their execution worker slot and suspend execution while waiting for external asynchronous operations (such as long-running BigQuery batch query jobs) to finish. The lightweight triggerer process monitors the external job status using asynchronous Python event loops, freeing worker capacity for other tasks. How It Meets the Requirements Orchestrating Complex Dependencies in Python: Cloud Composer utilizes declarative Directed Acyclic Graphs (DAGs) defined entirely in Python, allowing rich dependency structures, dynamic task generation, and granular retry policies. Preventing Worker Starvation: Configuring deferrable=True on operators like BigQueryInsertJobOperator ensures that the Airflow worker thread is released back to the worker pool while BigQuery executes the batch query asynchronously. Ensuring Idempotency: Utilizing SQL MERGE statements targeted at partitioned BigQuery tables guarantees deterministic upsert logic, ensuring identical outputs regardless of task retry counts. Automated Data Lineage: Cloud Composer natively integrates with Dataplex Universal Catalog via the apache-airflow-providers-openlineage provider, automatically tracking inputs and outputs of BigQueryInsertJobOperator without custom code. Key Advantages and Capabilities Optimized Resource Utilization: Eliminates idle worker threads blocking on long-running BigQuery queries. Out-of-the-Box Lineage: Automatically registers table-level and column-level lineage within Dataplex. Declarative Scalability: Scales Airflow workers dynamically according to queued workload volume. Robust Error Recovery: Native Airflow retry policies handle transient API and infrastructure failures cleanly. Why This Is Optimal This architecture pairs the asynchronous scalability of Airflow deferrable operators with idempotent MERGE query patterns and automated Dataplex governance, fulfilling all scaling, resilience, and operational observability criteria.

StudyCerts

Professional Cloud Data Engineer

professional-cloud-data-engineer

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Orchestrate Hybrid Batch Interactive Cloud Composer

Section: Interactive or batch query jobs

Your enterprise data team is architecting a multi-step data processing pipeline on Google Cloud that orchestrates dependencies between long-running BigQuery batch transformation jobs and interactive analytical workloads. The design must satisfy the following technical and operational requirements:

Orchestrate complex task dependencies, retries, and data transformations across hybrid services using Python.
Prevent worker resource starvation and task timeouts while waiting for long-running BigQuery batch jobs to complete.
Ensure end-to-end idempotency so that automated task retries or scheduled pipeline re-executions do not result in duplicate records or inconsistent states.
Automatically capture and publish end-to-end data lineage to Dataplex Universal Catalog without requiring custom reporting code.

Which orchestration and pipeline architecture should you implement?

Deploy a Cloud Composer environment with Airflow Triggerers enabled; define a DAG using deferrable operators like BigQueryInsertJobOperator (deferrable=True) to execute MERGE statements against partitioned BigQuery tables, and enable Dataplex data lineage integration.

Implement Google Cloud Workflows with standard HTTP connectors to trigger synchronous BigQuery query jobs with INSERT statements, and invoke the Dataplex Lineage REST API manually upon task completion.

Implement Cloud Scheduler to invoke a Cloud Dataflow batch pipeline that reads unpartitioned staging tables and uses TRUNCATE write dispositions to overwrite target BigQuery tables on each scheduled run.

Deploy a Cloud Composer DAG using standard synchronous PythonOperator tasks that execute continuous time.sleep polling loops against BigQuery batch jobs, appending records into unpartitioned staging tables.

Professional Cloud Data Engineer

professional-cloud-data-engineer

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Orchestrate Hybrid Batch Interactive Cloud Composer

Section: Interactive or batch query jobs

Orchestrate complex task dependencies, retries, and data transformations across hybrid services using Python.
Prevent worker resource starvation and task timeouts while waiting for long-running BigQuery batch jobs to complete.
Ensure end-to-end idempotency so that automated task retries or scheduled pipeline re-executions do not result in duplicate records or inconsistent states.
Automatically capture and publish end-to-end data lineage to Dataplex Universal Catalog without requiring custom reporting code.

Which orchestration and pipeline architecture should you implement?

Explanations

What Cloud Composer with Deferrable Operators Is

Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. By incorporating Airflow Triggerers and deferrable operators, Cloud Composer enables tasks to yield their execution worker slot and suspend execution while waiting for external asynchronous operations (such as long-running BigQuery batch query jobs) to finish. The lightweight triggerer process monitors the external job status using asynchronous Python event loops, freeing worker capacity for other tasks.

How It Meets the Requirements

Orchestrating Complex Dependencies in Python: Cloud Composer utilizes declarative Directed Acyclic Graphs (DAGs) defined entirely in Python, allowing rich dependency structures, dynamic task generation, and granular retry policies.
Preventing Worker Starvation: Configuring deferrable=True on operators like BigQueryInsertJobOperator ensures that the Airflow worker thread is released back to the worker pool while BigQuery executes the batch query asynchronously.
Ensuring Idempotency: Utilizing SQL MERGE statements targeted at partitioned BigQuery tables guarantees deterministic upsert logic, ensuring identical outputs regardless of task retry counts.
Automated Data Lineage: Cloud Composer natively integrates with Dataplex Universal Catalog via the apache-airflow-providers-openlineage provider, automatically tracking inputs and outputs of BigQueryInsertJobOperator without custom code.

Key Advantages and Capabilities

Optimized Resource Utilization: Eliminates idle worker threads blocking on long-running BigQuery queries.
Out-of-the-Box Lineage: Automatically registers table-level and column-level lineage within Dataplex.
Declarative Scalability: Scales Airflow workers dynamically according to queued workload volume.
Robust Error Recovery: Native Airflow retry policies handle transient API and infrastructure failures cleanly.

Why This Is Optimal

This architecture pairs the asynchronous scalability of Airflow deferrable operators with idempotent MERGE query patterns and automated Dataplex governance, fulfilling all scaling, resilience, and operational observability criteria.

View source documentation

Professional Cloud Data Engineer

Orchestrate Hybrid Batch Interactive Cloud Composer

Professional Cloud Data Engineer

Orchestrate Hybrid Batch Interactive Cloud Composer

What Cloud Composer with Deferrable Operators Is

How It Meets the Requirements

Key Advantages and Capabilities

Why This Is Optimal

Related questions