A data engineering team is architecting two interdependent workflows in Google Cloud Composer: an upstream ingestion workflow (DAG_Ingest) and a downstream transformation workflow (DAG_Analytics). The team must fulfill the following architectural and operational requirements: DAG_Analytics must begin processing only after DAG_Ingest completes and publishes an event artifact notification. Prevent automated scheduling of historical backfill runs when DAG_Analytics is uploaded or unpaused. Prevent resource starvation of core Airflow components (schedulers and web servers) caused by CPU- and memory-heavy transformation tasks. Ensure robust task failure recovery without whole-workflow failure. Which combination of Airflow configurations, operators, and task relationships should the team implement?

Configure `DAG_Analytics` with a static `start_date` and `catchup=False`. Implement a sensor in `DAG_Analytics` to trigger execution on the event notification, execute heavy tasks via `KubernetesPodOperator` or `GKEStartPodOperator`, and define `retries` and `retry_delay` in `default_args`.: ### What this architectural solution is This architecture combines event-driven triggering, proper DAG scheduling hygiene, isolated compute execution, and robust error recovery mechanisms in **Cloud Composer**. ### How it meets the requirements - **Decoupled Event-Driven Dependency**: Using a sensor (such as an event sensor or `PubSubPullSensor`) at the start of `DAG_Analytics` allows the downstream workflow to wait for an artifact or notification emitted by `DAG_Ingest` without requiring tight inter-DAG operator coupling. - **Preventing Historical Backfills**: Setting `catchup=False` alongside a fixed, static `start_date` ensures Airflow only schedules DAG runs for the current/future intervals and does not spawn unwanted historical runs when unpaused or parsed. - **Isolating Heavy Compute**: Using **`KubernetesPodOperator`** or **`GKEStartPodOperator`** executes heavy data transformations in separate Kubernetes pods or dedicated GKE node pools, preventing resource starvation of the core Airflow scheduler, webserver, and worker processes. - **Robust Error Recovery**: Setting `retries` (greater than 0) and a `retry_delay` inside `default_args` ensures transient task failures retry automatically instead of failing the entire DAG run. ### Key advantages - **Component Stability**: Offloading intensive compute preserves the stability of the Composer control plane and worker nodes. - **Predictable Scheduling**: Disabling `catchup` avoids scheduler overload caused by hundreds of backfill runs queuing simultaneously. - **Idempotency & Resilience**: Built-in retries handle intermittent network or downstream API hiccups gracefully. ### Why this is optimal Decoupling workflows using sensors and offloading execution to containerized compute pools follows Google Cloud recommended best practices for enterprise-grade orchestration.

Cloud Composer Dag Architecture Dependencies | professional-cloud-data-engineer

Cloud Composer Dag Architecture Dependencies

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A data engineering team is architecting two interdependent workflows in Google Cloud Composer: an upstream ingestion workflow (DAG_Ingest) and a downstream transformation workflow (DAG_Analytics).

The team must fulfill the following architectural and operational requirements:

DAG_Analytics must begin processing only after DAG_Ingest completes and publishes an event artifact notification.
Prevent automated scheduling of historical backfill runs when DAG_Analytics is uploaded or unpaused.
Prevent resource starvation of core Airflow components (schedulers and web servers) caused by CPU- and memory-heavy transformation tasks.
Ensure robust task failure recovery without whole-workflow failure.

Which combination of Airflow configurations, operators, and task relationships should the team implement?

Configure DAG_Analytics with a static start_date and catchup=False. Implement a sensor in DAG_Analytics to trigger execution on the event notification, execute heavy tasks via KubernetesPodOperator or GKEStartPodOperator, and define retries and retry_delay in default_args.

Configure DAG_Analytics with a dynamic start_date=datetime.now() and catchup=True. Use a TriggerDagRunOperator at the end of DAG_Ingest, run transformation tasks directly within local PythonOperator callables, and set task retries=0.

Encapsulate DAG_Analytics inside a SubDagOperator within DAG_Ingest. Keep catchup=True with dynamic start dates, execute transformations in local BashOperator tasks on Composer workers, and set task retries to 0.

Place custom transformation scripts in the /data folder. Trigger DAG_Analytics via periodic polling loops inside a DummyOperator, configure catchup=True, and offload compute using standard Airflow worker threads without retry policies.

Professional Cloud Data Engineer

Cloud Composer Dag Architecture Dependencies

Related questions