Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team manages a daily batch pipeline orchestrated by Cloud Composer that processes terabytes of transactional log files from Cloud Storage into partitioned BigQuery tables. Recently, transient worker pod evictions and network timeouts have triggered task retries, resulting in duplicate data rows in BigQuery and failed task executions caused by exhausted worker disk space.
You need to re-architect the workflow orchestration to ensure fault tolerance, prevent duplicate records during task retries (idempotency), and prevent worker resource exhaustion.
Which architectural strategy should you implement?
BigQueryExecuteQueryOperator to stage records into the Airflow metadata database, then invoke the Airflow Web Server API to trigger manual transaction rollbacks upon task failure.This approach suggests using deprecated Google Cloud operators to route transactional batch data through the Airflow Cloud SQL metadata database and using Web Server API calls to manage transaction state.
BigQueryExecuteQueryOperator is deprecated and slated for removal in modern Airflow provider packages.Cloud SQL) is strictly designed for orchestration state and DAG metadata, with strict sizing guidelines (keeping database size below 20 GB). Staging batch data inside Cloud SQL causes critical database locks, performance collapse, and environment failures.The scenario asked about legacy Airflow 1 architecture components and small transactional metadata logging rather than scalable batch pipeline orchestration.
WRITE_TRUNCATE or execute SQL MERGE statements, load data directly from Cloud Storage to BigQuery without staging on local worker disks, and configure declarative task retries with exponential backoff.This architecture establishes an idempotent, serverless batch ELT pipeline managed by Cloud Composer that offloads heavy compute and storage operations to BigQuery and Cloud Storage, while leveraging declarative Airflow retry mechanisms.
WRITE_TRUNCATE targeting a specific partition decorator (e.g., table$20260330) or executing atomic SQL MERGE statements guarantees that re-running a failed or evicted task completely overwrites or reconciles the specific partition, eliminating duplicate records upon retry.default_args allows Airflow to cleanly recover from transient errors without human intervention.retries, retry_delay, retry_exponential_backoff).Batch pipelines must decouple orchestration from data execution. Cloud Composer functions optimally as an orchestrator rather than a heavy execution engine. Offloading batch transforms to BigQuery and configuring atomic partition-level overwrites ensures exact-once semantics and maximum pipeline resilience.
WRITE_APPEND while tracking retry state via top-level Variable.get calls.This option attempts to stage files directly on the worker node's local filesystem, perform in-memory deduplication using Python inside worker containers, and log state using Airflow Variables in the top-level script before appending records to BigQuery.
OOMKilling / Negsignal.SIGKILL.WRITE_APPEND creates duplicate rows whenever a task fails mid-execution and is subsequently retried by the scheduler.Variable.get in top-level DAG code creates severe connection bottlenecks on the Airflow Cloud SQL metadata database during regular DAG parsing intervals.WRITE_APPEND blindly appends incoming rows without checking for existing keys.The workload consisted of tiny configuration metadata files (<10 KB) requiring local parsing without any database insertion or heavy throughput requirements.
WRITE_APPEND mode and attach an on_failure_callback that executes a downstream script to scan and delete all duplicate records across the entire BigQuery table if any failure occurs.This option relies on appending batch data blindly and using Airflow's on_failure_callback hook to trigger a full table scan that identifies and deletes duplicates when a task fails.
SIGKILL), the on_failure_callback might not execute reliably or clean up partially written append batches.The destination storage system did not support partitioned atomic writes, transactions, or merge semantics, and cleanup had to be handled via external compensating actions.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated