A data engineering team is deploying a critical daily data pipeline in Google Cloud Composer. The pipeline extracts transactional records, transforms them, and writes them to BigQuery. The pipeline must satisfy the following operational resilience and error handling requirements: If a worker node is interrupted during environment maintenance or encounters a transient network glitch, running tasks must automatically attempt recovery with progressively increasing intervals between attempts. A task in the current DAG run must not execute if the immediate downstream tasks from the previous DAG run have not yet succeeded. The team must be alerted and a cleanup routine must be executed only after a task has completely exhausted all configured retry attempts. Which configuration should you implement in the DAG definition?

Define default_args with retries and retry_exponential_backoff=True, set wait_for_downstream=True on the relevant tasks, and configure the on_failure_callback parameter with an alerting and cleanup function.: ### What this configuration is This configuration leverages **Apache Airflow's built-in resilience and callback mechanisms** within **Cloud Composer** to manage task retries, cross-run dependencies, and terminal failure handling. ### How it meets the requirements - **Progressive recovery intervals**: Setting `retries` alongside `retry_exponential_backoff=True` ensures that if a task fails or is interrupted during **Cloud Composer maintenance windows**, Airflow retries the task while exponentially increasing the delay (`retry_delay`) between attempts to prevent hammering transiently degraded systems. - **Preventing execution on previous run downstream failures**: Setting `wait_for_downstream=True` enforces that a task in DAG run *N* cannot run until all tasks immediately downstream of that same task in DAG run *N-1* have succeeded, maintaining strict pipeline integrity across consecutive runs. - **Terminal failure alerting and cleanup**: The `on_failure_callback` hook triggers exclusively when a task reaches a terminal **failed** state (after all allotted `retries` have been exhausted), preventing premature alert noise during intermediate retries. ### Key advantages - **Resilience during maintenance**: Handles unexpected worker evictions or infrastructure maintenance automatically. - **Cross-run data integrity**: Avoids corrupting data models by enforcing strict dependency verification across DAG runs. - **Targeted alerting**: Restricts PagerDuty/Slack alerts and rollback scripts to true unrecoverable failures. - **Backoff protection**: Prevents resource starvation by spacing out subsequent execution attempts. ### Why this is optimal This approach uses standard, declarative DAG parameters without requiring custom scheduling loops or external monitoring agents, providing a robust operational error handling posture.

Composer Dag Error Handling Resilience | professional-cloud-data-engineer

Composer Dag Error Handling Resilience

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A data engineering team is deploying a critical daily data pipeline in Google Cloud Composer. The pipeline extracts transactional records, transforms them, and writes them to BigQuery.

The pipeline must satisfy the following operational resilience and error handling requirements:

If a worker node is interrupted during environment maintenance or encounters a transient network glitch, running tasks must automatically attempt recovery with progressively increasing intervals between attempts.
A task in the current DAG run must not execute if the immediate downstream tasks from the previous DAG run have not yet succeeded.
The team must be alerted and a cleanup routine must be executed only after a task has completely exhausted all configured retry attempts.

Which configuration should you implement in the DAG definition?

Configure wait_for_downstream=True, set catchup=True with a dynamic start_date=datetime.now(), and configure the sla_miss_callback parameter with an alerting function.

Define default_args with retries and retry_exponential_backoff=True, set depends_on_past=True on the relevant tasks, and configure the on_retry_callback parameter with an alerting and cleanup function.

Adjust [scheduler]task_queued_timeout to match [core]parallelism, set depends_on_past=False, and configure the on_execute_callback parameter with an alerting function.

Define default_args with retries and retry_exponential_backoff=True, set wait_for_downstream=True on the relevant tasks, and configure the on_failure_callback parameter with an alerting and cleanup function.

Professional Cloud Data Engineer

Composer Dag Error Handling Resilience

Related questions