professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.
Last updated
Your enterprise data team is architecting a multi-step data processing pipeline on Google Cloud that orchestrates dependencies between long-running BigQuery batch transformation jobs and interactive analytical workloads. The design must satisfy the following technical and operational requirements:
Which orchestration and pipeline architecture should you implement?
Deploy a Cloud Composer environment with Airflow Triggerers enabled; define a DAG using deferrable operators like BigQueryInsertJobOperator (deferrable=True) to execute MERGE statements against partitioned BigQuery tables, and enable Dataplex data lineage integration.
Implement Cloud Scheduler to invoke a Cloud Dataflow batch pipeline that reads unpartitioned staging tables and uses TRUNCATE write dispositions to overwrite target BigQuery tables on each scheduled run.
Deploy a Cloud Composer DAG using standard synchronous PythonOperator tasks that execute continuous time.sleep polling loops against BigQuery batch jobs, appending records into unpartitioned staging tables.
Implement Google Cloud Workflows with standard HTTP connectors to trigger synchronous BigQuery query jobs with INSERT statements, and invoke the Dataplex Lineage REST API manually upon task completion.
Deploy a Cloud Composer environment with Airflow Triggerers enabled; define a DAG using deferrable operators like BigQueryInsertJobOperator (deferrable=True) to execute MERGE statements against partitioned BigQuery tables, and enable Dataplex data lineage integration.
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. By incorporating Airflow Triggerers and deferrable operators, Cloud Composer enables tasks to yield their execution worker slot and suspend execution while waiting for external asynchronous operations (such as long-running BigQuery batch query jobs) to finish. The lightweight triggerer process monitors the external job status using asynchronous Python event loops, freeing worker capacity for other tasks.
StudyCerts
StudyCertsdeferrable=True on operators like BigQueryInsertJobOperator ensures that the Airflow worker thread is released back to the worker pool while BigQuery executes the batch query asynchronously.MERGE statements targeted at partitioned BigQuery tables guarantees deterministic upsert logic, ensuring identical outputs regardless of task retry counts.apache-airflow-providers-openlineage provider, automatically tracking inputs and outputs of BigQueryInsertJobOperator without custom code.This architecture pairs the asynchronous scalability of Airflow deferrable operators with idempotent MERGE query patterns and automated Dataplex governance, fulfilling all scaling, resilience, and operational observability criteria.
Implement Cloud Scheduler to invoke a Cloud Dataflow batch pipeline that reads unpartitioned staging tables and uses TRUNCATE write dispositions to overwrite target BigQuery tables on each scheduled run.
Deploy a Cloud Composer DAG using standard synchronous PythonOperator tasks that execute continuous time.sleep polling loops against BigQuery batch jobs, appending records into unpartitioned staging tables.
Implement Google Cloud Workflows with standard HTTP connectors to trigger synchronous BigQuery query jobs with INSERT statements, and invoke the Dataplex Lineage REST API manually upon task completion.