Cloud Composer Batch Idempotent Pipeline Design

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A data engineering team manages a daily batch pipeline orchestrated by Cloud Composer that processes terabytes of transactional log files from Cloud Storage into partitioned BigQuery tables. Recently, transient worker pod evictions and network timeouts have triggered task retries, resulting in duplicate data rows in BigQuery and failed task executions caused by exhausted worker disk space.

You need to re-architect the workflow orchestration to ensure fault tolerance, prevent duplicate records during task retries (idempotency), and prevent worker resource exhaustion.

Which architectural strategy should you implement?

Configure tasks with WRITE_APPEND mode and attach an on_failure_callback that executes a downstream script to scan and delete all duplicate records across the entire BigQuery table if any failure occurs.

Download the daily Cloud Storage files into the Airflow worker's local storage directory, deduplicate the records in Python worker memory, and insert records into BigQuery using WRITE_APPEND while tracking retry state via top-level Variable.get calls.

Configure BigQuery jobs to write to specific partition decorators using WRITE_TRUNCATE or execute SQL MERGE statements, load data directly from Cloud Storage to BigQuery without staging on local worker disks, and configure declarative task retries with exponential backoff.

Use deprecated Airflow operators like BigQueryExecuteQueryOperator to stage records into the Airflow metadata database, then invoke the Airflow Web Server API to trigger manual transaction rollbacks upon task failure.

Professional Cloud Data Engineer

Cloud Composer Batch Idempotent Pipeline Design

Related questions