Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team is deploying a critical daily data pipeline in Google Cloud Composer. The pipeline extracts transactional records, transforms them, and writes them to BigQuery.
The pipeline must satisfy the following operational resilience and error handling requirements:
Which configuration should you implement in the DAG definition?
This configuration modifies low-level Airflow scheduler configurations (task_queued_timeout and parallelism) and attaches a callback function to the task execution start event.
[scheduler]task_queued_timeout dictates the threshold in seconds before a stalled queued task is rescheduled, while [core]parallelism limits maximum concurrent task instances. Setting them equal to each other makes no operational sense.depends_on_past=False disables dependency validation between successive DAG runs.on_execute_callback fires when the task begins running, not when it fails after exhausting retries.on_execute_callback runs at task startup rather than on error.You needed to log an audit event whenever a task successfully acquired an Airflow worker slot and began processing.
This configuration applies standard exponential backoff retries, enables the depends_on_past flag to check the previous run of the same task, and triggers an on_retry_callback function.
depends_on_past=True only checks whether the exact same task in the previous DAG run (N-1) succeeded. It completely ignores whether downstream child tasks in the previous run succeeded or failed.on_retry_callback executes every single time a task enters an up_for_retry state. This generates alerts on intermediate transient failures rather than waiting until all retries are exhausted.depends_on_past is blind to downstream lineage failures.on_retry_callback causes alert fatigue during transient network blips.You needed to notify on-call engineers at the first sign of an interruption and only needed to verify that the single upstream task itself had completed previously.
This configuration combines downstream dependency checking with dynamic DAG start dates (datetime.now()) and Airflow Service Level Agreement (SLA) miss callbacks.
datetime.now() as a start_date causes severe scheduling bugs in Cloud Composer, leading to DAG runs never executing or executing in unexpected future intervals.retries and retry_exponential_backoff configurations for handling task interruptions during maintenance.sla_miss_callback is designed for timing thresholds (when tasks take longer than expected), not for triggering cleanup scripts upon terminal task failure.The primary goal was monitoring query execution duration against SLA targets on statically scheduled DAGs.
This configuration leverages Apache Airflow's built-in resilience and callback mechanisms within Cloud Composer to manage task retries, cross-run dependencies, and terminal failure handling.
retries alongside retry_exponential_backoff=True ensures that if a task fails or is interrupted during Cloud Composer maintenance windows, Airflow retries the task while exponentially increasing the delay (retry_delay) between attempts to prevent hammering transiently degraded systems.wait_for_downstream=True enforces that a task in DAG run N cannot run until all tasks immediately downstream of that same task in DAG run N-1 have succeeded, maintaining strict pipeline integrity across consecutive runs.on_failure_callback hook triggers exclusively when a task reaches a terminal failed state (after all allotted retries have been exhausted), preventing premature alert noise during intermediate retries.This approach uses standard, declarative DAG parameters without requiring custom scheduling loops or external monitoring agents, providing a robust operational error handling posture.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated