professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.
Last updated
A data engineering team is architecting two interdependent workflows in Google Cloud Composer: an upstream ingestion workflow (DAG_Ingest) and a downstream transformation workflow (DAG_Analytics).
The team must fulfill the following architectural and operational requirements:
StudyCerts
StudyCertsDAG_Analytics must begin processing only after DAG_Ingest completes and publishes an event artifact notification.DAG_Analytics is uploaded or unpaused.Which combination of Airflow configurations, operators, and task relationships should the team implement?
Configure DAG_Analytics with a static start_date and catchup=False. Implement a sensor in DAG_Analytics to trigger execution on the event notification, execute heavy tasks via KubernetesPodOperator or GKEStartPodOperator, and define retries and retry_delay in default_args.
Encapsulate DAG_Analytics inside a SubDagOperator within DAG_Ingest. Keep catchup=True with dynamic start dates, execute transformations in local BashOperator tasks on Composer workers, and set task retries to 0.
Place custom transformation scripts in the /data folder. Trigger DAG_Analytics via periodic polling loops inside a DummyOperator, configure catchup=True, and offload compute using standard Airflow worker threads without retry policies.
Configure DAG_Analytics with a dynamic start_date=datetime.now() and catchup=True. Use a TriggerDagRunOperator at the end of DAG_Ingest, run transformation tasks directly within local PythonOperator callables, and set task retries=0.
Configure DAG_Analytics with a static start_date and catchup=False. Implement a sensor in DAG_Analytics to trigger execution on the event notification, execute heavy tasks via KubernetesPodOperator or GKEStartPodOperator, and define retries and retry_delay in default_args.
This architecture combines event-driven triggering, proper DAG scheduling hygiene, isolated compute execution, and robust error recovery mechanisms in Cloud Composer.
PubSubPullSensor) at the start of DAG_Analytics allows the downstream workflow to wait for an artifact or notification emitted by DAG_Ingest without requiring tight inter-DAG operator coupling.catchup=False alongside a fixed, static start_date ensures Airflow only schedules DAG runs for the current/future intervals and does not spawn unwanted historical runs when unpaused or parsed.KubernetesPodOperator or GKEStartPodOperator executes heavy data transformations in separate Kubernetes pods or dedicated GKE node pools, preventing resource starvation of the core Airflow scheduler, webserver, and worker processes.retries (greater than 0) and a retry_delay inside default_args ensures transient task failures retry automatically instead of failing the entire DAG run.catchup avoids scheduler overload caused by hundreds of backfill runs queuing simultaneously.Decoupling workflows using sensors and offloading execution to containerized compute pools follows Google Cloud recommended best practices for enterprise-grade orchestration.
Encapsulate DAG_Analytics inside a SubDagOperator within DAG_Ingest. Keep catchup=True with dynamic start dates, execute transformations in local BashOperator tasks on Composer workers, and set task retries to 0.
Place custom transformation scripts in the /data folder. Trigger DAG_Analytics via periodic polling loops inside a DummyOperator, configure catchup=True, and offload compute using standard Airflow worker threads without retry policies.
Configure DAG_Analytics with a dynamic start_date=datetime.now() and catchup=True. Use a TriggerDagRunOperator at the end of DAG_Ingest, run transformation tasks directly within local PythonOperator callables, and set task retries=0.