Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise data platform processes daily transactions and telemetry data across multiple business domains in Google Cloud. Several downstream analytical pipelines (Finance P&L, Regional Operations, and Customer Insights) depend on a core transformed dataset produced by an upstream Dataflow pipeline.
Under the existing monolithic orchestration structure, intermittent failures in one analytical branch cause cascading pipeline delays across unrelated domains. The data engineering team must re-architect the orchestration to satisfy the following requirements:
Which orchestration design should the team implement in Cloud Composer?
TaskGroup to visually organize internal steps, and use TriggerDagRunOperator with runtime parameters passed in the conf dictionary to trigger downstream domain DAGs.This architecture establishes a modular, decoupled orchestration design in Cloud Composer (Apache Airflow). By separating shared data processing from downstream domain-specific transformations into independent DAGs, the workflow leverages TaskGroup for UI organization and TriggerDagRunOperator for cross-DAG event-driven coordination.
TriggerDagRunOperator allows the controlling upstream DAG to invoke downstream DAG runs programmatically while supplying dynamic configuration dictionaries via the conf parameter (e.g., conf={"batch_id": "12345", "ds": "2024-01-01"}). Downstream tasks can access these values dynamically using the dag_run.conf context variable.TaskGroup enables visual and structural grouping of related tasks (such as ingestion validation checks) in the Airflow UI while keeping all tasks within the same DAG execution graph, avoiding the scheduling deadlocks and resource constraints associated with legacy sub-DAGs.TaskGroup simplifies complex DAG graphs in the Airflow web interface with expandable/collapsible UI containers without altering runtime execution mechanics.conf payload cleanly delivers execution metadata at trigger time without relying on persistent database writes or bloated XCom storage.Migrating monolithic workflows into modular DAGs connected via TriggerDagRunOperator represents the Google Cloud recommended best practice for orchestrating fan-out data warehousing pipelines. It provides the highest degree of fault tolerance, operational clarity, and decoupled scalability.
SubDagOperator blocks and transferring parameters between domains using Airflow XComs.This option retains a single monolithic DAG architecture and relies on the legacy SubDagOperator to group tasks into embedded sub-DAGs, using Airflow XCom (Cross-Communication) to pass state and parameters between execution nodes.
SubDagOperator within a single master DAG does not decouple them. A failure within one sub-DAG can still stall or fail the overall parent DAG execution, leading to cascading delays across unrelated business units.SubDagOperator creates a separate execution context with its own limited worker pool and scheduler constraints, frequently leading to executor deadlocks in Airflow environments.SubDagOperator in favor of TaskGroup due to known performance, scheduling, and visualization issues.You were maintaining an obsolete Airflow 1.x deployment where TaskGroup was not yet implemented and pipelines were strictly bound to a single synchronized release lifecycle.
ExternalTaskSensor instances configured in poke mode and passing execution variables via the global Airflow Variables table.This option keeps all processing branches in a single large DAG, synchronizing branches using ExternalTaskSensor with active polling (poke mode) and writing runtime execution parameters to the global Airflow Variables metadata store.
poke mode keeps worker slots continuously occupied while polling, which quickly starves the Cloud Composer worker pool of execution capacity during heavy data processing runs.ExternalTaskSensor misuse: ExternalTaskSensor is intended for cross-DAG synchronization across separate DAG schedules, not for coordinating branches within the same DAG.Downstream DAGs were running on distinct, fixed cron schedules and needed to poll for the completion of an upstream DAG without direct trigger-based coupling, using reschedule mode instead of poke mode.
This option replaces Cloud Composer with Google Cloud Workflows, attempting to define heavy data transformation pipelines and maintain cross-pipeline states directly within serverless YAML workflow definitions.
You needed a serverless, low-cost solution to orchestrate lightweight HTTP microservices and Cloud Run jobs with millisecond start times and low execution volume.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated