professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.
Last updated
A data engineering team manages a daily batch pipeline orchestrated by Cloud Composer that processes terabytes of transactional log files from Cloud Storage into partitioned BigQuery tables. Recently, transient worker pod evictions and network timeouts have triggered task retries, resulting in duplicate data rows in BigQuery and failed task executions caused by exhausted worker disk space.
You need to re-architect the workflow orchestration to ensure fault tolerance, prevent duplicate records during task retries (idempotency), and prevent worker resource exhaustion.
Which architectural strategy should you implement?
Configure tasks with WRITE_APPEND mode and attach an on_failure_callback that executes a downstream script to scan and delete all duplicate records across the entire BigQuery table if any failure occurs.
Use deprecated Airflow operators like BigQueryExecuteQueryOperator to stage records into the Airflow metadata database, then invoke the Airflow Web Server API to trigger manual transaction rollbacks upon task failure.
Download the daily Cloud Storage files into the Airflow worker's local storage directory, deduplicate the records in Python worker memory, and insert records into BigQuery using WRITE_APPEND while tracking retry state via top-level Variable.get calls.
Configure BigQuery jobs to write to specific partition decorators using WRITE_TRUNCATE or execute SQL MERGE statements, load data directly from Cloud Storage to BigQuery without staging on local worker disks, and configure declarative task retries with exponential backoff.
Configure tasks with WRITE_APPEND mode and attach an on_failure_callback that executes a downstream script to scan and delete all duplicate records across the entire BigQuery table if any failure occurs.
Use deprecated Airflow operators like BigQueryExecuteQueryOperator to stage records into the Airflow metadata database, then invoke the Airflow Web Server API to trigger manual transaction rollbacks upon task failure.
Download the daily Cloud Storage files into the Airflow worker's local storage directory, deduplicate the records in Python worker memory, and insert records into BigQuery using WRITE_APPEND while tracking retry state via top-level Variable.get calls.
Configure BigQuery jobs to write to specific partition decorators using WRITE_TRUNCATE or execute SQL MERGE statements, load data directly from Cloud Storage to BigQuery without staging on local worker disks, and configure declarative task retries with exponential backoff.
This architecture establishes an idempotent, serverless batch ELT pipeline managed by Cloud Composer that offloads heavy compute and storage operations to BigQuery and Cloud Storage, while leveraging declarative Airflow retry mechanisms.
StudyCerts
StudyCertsWRITE_TRUNCATE targeting a specific partition decorator (e.g., table$20260330) or executing atomic SQL MERGE statements guarantees that re-running a failed or evicted task completely overwrites or reconciles the specific partition, eliminating duplicate records upon retry.default_args allows Airflow to cleanly recover from transient errors without human intervention.retries, retry_delay, retry_exponential_backoff).Batch pipelines must decouple orchestration from data execution. Cloud Composer functions optimally as an orchestrator rather than a heavy execution engine. Offloading batch transforms to BigQuery and configuring atomic partition-level overwrites ensures exact-once semantics and maximum pipeline resilience.