Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team is architecting two interdependent workflows in Google Cloud Composer: an upstream ingestion workflow (DAG_Ingest) and a downstream transformation workflow (DAG_Analytics).
The team must fulfill the following architectural and operational requirements:
DAG_Analytics must begin processing only after DAG_Ingest completes and publishes an event artifact notification.DAG_Analytics is uploaded or unpaused.Which combination of Airflow configurations, operators, and task relationships should the team implement?
DAG_Analytics with a static start_date and catchup=False. Implement a sensor in DAG_Analytics to trigger execution on the event notification, execute heavy tasks via KubernetesPodOperator or GKEStartPodOperator, and define retries and retry_delay in default_args.This architecture combines event-driven triggering, proper DAG scheduling hygiene, isolated compute execution, and robust error recovery mechanisms in Cloud Composer.
PubSubPullSensor) at the start of DAG_Analytics allows the downstream workflow to wait for an artifact or notification emitted by DAG_Ingest without requiring tight inter-DAG operator coupling.catchup=False alongside a fixed, static start_date ensures Airflow only schedules DAG runs for the current/future intervals and does not spawn unwanted historical runs when unpaused or parsed.KubernetesPodOperator or GKEStartPodOperator executes heavy data transformations in separate Kubernetes pods or dedicated GKE node pools, preventing resource starvation of the core Airflow scheduler, webserver, and worker processes.retries (greater than 0) and a retry_delay inside default_args ensures transient task failures retry automatically instead of failing the entire DAG run.catchup avoids scheduler overload caused by hundreds of backfill runs queuing simultaneously.Decoupling workflows using sensors and offloading execution to containerized compute pools follows Google Cloud recommended best practices for enterprise-grade orchestration.
/data folder. Trigger DAG_Analytics via periodic polling loops inside a DummyOperator, configure catchup=True, and offload compute using standard Airflow worker threads without retry policies.This pattern places dependent libraries inside the Cloud Storage /data directory, utilizes DummyOperator for custom polling logic, and executes workloads across default worker threads without task-level retry handling.
/dags folder or within custom packaged libraries, not in the /data directory (which is intended for generic file storage).DummyOperator (or EmptyOperator) performs no work and cannot poll message queues or sensors for event triggers.catchup=True triggers historical runs, and omitting retries leaves the workflow vulnerable to transient infrastructure errors./data.The goal was simply to store arbitrary data artifacts in Cloud Storage without orchestrating automated execution or managing task dependencies.
DAG_Analytics inside a SubDagOperator within DAG_Ingest. Keep catchup=True with dynamic start dates, execute transformations in local BashOperator tasks on Composer workers, and set task retries to 0.This approach uses Airflow's legacy SubDagOperator to embed the transformation tasks inside the ingestion DAG while executing shell scripts directly on the Airflow worker infrastructure.
SubDagOperator is heavily discouraged and deprecated in modern Airflow versions due to deadlocks, scheduler bottlenecks, and poor observability. Modern architectures use TaskGroup instead.BashOperator directly on worker instances starves Composer system daemons.catchup=True executes backfill runs for all historical execution intervals.0 disables failure recovery.SubDagOperator monopolizes worker execution slots and can cause DAG deadlocks.You were working on a legacy Airflow 1.x pipeline specifically designed around monolithic subDAG execution with strictly no retry tolerance.
DAG_Analytics with a dynamic start_date=datetime.now() and catchup=True. Use a TriggerDagRunOperator at the end of DAG_Ingest, run transformation tasks directly within local PythonOperator callables, and set task retries=0.This configuration relies on dynamic start dates, tight upstream-to-downstream coupling using TriggerDagRunOperator, executes compute directly within Airflow worker processes via PythonOperator, and disables retries.
datetime.now() evaluates dynamically every time the DAG file is parsed by the Airflow scheduler, creating severe scheduling anomalies and unexpected task instance states.PythonOperator runs code directly on the shared Composer worker nodes, risking memory exhaustion (OOM) and scheduler crashes.catchup=True triggers past runs if the start date falls behind the current execution timestamp.retries=0 causes any transient task failure to immediately fail the entire DAG run.dag_id.This setup would only be acceptable for lightweight, non-critical scripts where immediate fail-fast behavior is required and no historical runs could ever exist.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated