Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team is orchestrating a multi-step ETL pipeline in Google Cloud Composer using Apache Airflow. The workflow includes conditional execution logic:
BranchPythonOperator to determine if new source records are present in Cloud Storage.During testing, when no records exist, the logging task succeeds, but the downstream summary notification task is automatically marked as skipped and never runs.
How should the team modify the DAG configuration to ensure the summary notification task executes as required?
BranchPythonOperator to return the task IDs of both branches simultaneously and enable depends_on_past=True on the downstream taskReturning multiple task IDs from a BranchPythonOperator directs Airflow to execute both downstream paths concurrently instead of conditionally choosing one, while depends_on_past=True forces a task instance to wait for the exact same task in the prior DAG run to succeed.
depends_on_past=True does not resolve the intra-run trigger rule mismatch; if a previous run was skipped or failed, subsequent runs of the task will be blocked indefinitely.BranchPythonOperator is designed to return a single target task ID (or a specific subset) to skip non-selected paths.depends_on_past controls sequential run scheduling across time, not inter-task dependencies within a single DAG run.You needed sequential time-series backfills where a task must only run if yesterday's execution of that exact same task completed successfully.
TriggerRule.ALL_SUCCESSA SubDAG (or SubDagOperator) encapsulates a subordinate DAG workflow inside a parent task node, presenting a single task representation to the primary DAG schedule.
all_success skip behavior unless trigger rules are altered.SubDagOperator is deprecated in modern Airflow and Cloud Composer due to known deadlocking, executor resource starvation, and performance bottlenecks, having been replaced by TaskGroup.SubDAGs create separate execution contexts that consume independent scheduler and worker slots.You were maintaining a legacy Airflow 1.x pipeline requiring UI grouping and modular execution blocks prior to the introduction of TaskGroup.
trigger_rule parameter on the summary notification task to TriggerRule.NONE_FAILED_MIN_ONE_SUCCESSTriggerRule.NONE_FAILED_MIN_ONE_SUCCESS isIn Apache Airflow orchestrations within Cloud Composer, task execution dependencies are governed by trigger rules. By default, tasks use the all_success trigger rule, which requires every direct upstream parent task to reach the success state before downstream tasks can start.
BranchPythonOperator selects a path, unselected branch tasks are placed in a skipped state.TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS, the downstream summary notification task will trigger as long as all upstream parent tasks have finished, no upstream task has failed (failed or upstream_failed), and at least one direct parent task succeeded.This approach directly solves the default Airflow cascade skip behavior associated with branching operators without compromising failure detection or adding redundant operator overhead.
trigger_rule parameter on the summary notification task to TriggerRule.ALL_FAILED and remove upstream task linkagesThe TriggerRule.ALL_FAILED setting instructs Airflow to execute a task only when every direct upstream parent task has failed, and removing task linkages detaches dependency tracking entirely.
TriggerRule.ALL_FAILED is intended for failure remediation, alerting, or cleanup tasks that should run only upon complete pipeline failure.>> or set_upstream/set_downstream) define the directed acyclic graph structure; removing them eliminates deterministic execution ordering.You were designing a dedicated dead-letter alerting task intended to fire exclusively when all extraction and logging attempts crashed.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated