Fundamental
Prepare and test your skills
Fundamental
Prepare and test your skills
Cloud Composer DAG Dependencies for Sequential BigQuery Tasks
Free sample question, answered and explained below. The Associate Data Practitioner practice bank has 1,132 more like it. Try 10 free: instant score, no account.
A data engineering team is designing an analytics workflow in Google Cloud that consists of several dependent steps:
- Ingesting raw logs into a BigQuery staging table
- Executing SQL aggregation transformations on the staging table to update summary tables
- Running data quality verification checks and publishing completion status
To prevent race conditions, each step must execute only after the previous step completes successfully rather than relying on estimated time delays.
How should the team implement this workflow using Cloud Composer?
Explanations
Configure independent BigQuery scheduled queries for each task, setting staggered execution start times across the hour.
Configure independent BigQuery scheduled queries for each task, setting staggered execution start times across the hour.
What BigQuery scheduled queries do
BigQuery scheduled queries allow users to automate recurring SQL queries on a recurring schedule (for example, every hour or every day at midnight). Each scheduled query runs independently as an isolated BigQuery job managed natively within the BigQuery service.
Why it doesn't meet the requirement
- No dependency awareness (✗): Scheduled queries are unaware of the success or failure of other queries. Staggering start times (e.g., running query 1 at 1:00 AM, query 2 at 1:15 AM) relies on time estimates rather than actual task completion.
- Risk of race conditions (✗): If data ingestion takes longer than anticipated (e.g., due to higher data volumes or platform throttling), the aggregation query will execute prematurely on incomplete data.
- Silent data corruption: If the staging task fails completely, downstream scheduled queries will still execute and aggregate outdated data, producing inaccurate reports.
Key characteristics and limitations
- Isolated execution: Each query schedule operates in a silo without inter-job signaling.
- Simple time triggers only: Best suited for standalone, independent transformation queries that do not depend on complex multi-step pipelines.
- Limited orchestration features: Lacks built-in branching, conditional logic, cross-service task triggers, or dynamic retries.
This answer would be correct if
The requirement involved running a single, self-contained SQL aggregation query on an existing dataset once per day where no upstream or downstream dependencies needed to be coordinated.
Set up individual Cloud Scheduler jobs with cron expressions to invoke the BigQuery API for each step at fixed intervals.
Set up individual Cloud Scheduler jobs with cron expressions to invoke the BigQuery API for each step at fixed intervals.
What Cloud Scheduler does
Cloud Scheduler is a fully managed enterprise cron job service. It allows you to trigger target endpoints (such as HTTP endpoints, Pub/Sub topics, or App Engine services) on recurring time schedules using standard cron syntax.
Why it doesn't meet the requirement
- Lacks workflow orchestration logic (✗): Cloud Scheduler is a point-in-time event generator, not a workflow engine. It cannot track task execution states or dynamically trigger a downstream job upon the successful completion of an upstream job.
- Prone to timing failures (✗): Just like scheduled queries, using Cloud Scheduler with fixed time intervals assumes every step finishes within an allocated time window, leading to race conditions if any step takes longer than expected.
- Complex error management: Cloud Scheduler cannot automatically hold or skip downstream tasks when an earlier job fails.
Key characteristics and limitations
- Cron-based triggering: Triggers events strictly according to defined clock intervals.
- Stateless execution: Does not maintain state across multiple sequential operations.
- Target invocation: Commonly used to ping a webhook, send a Pub/Sub message, or trigger a Cloud Function rather than coordinating complex DAGs.
This answer would be correct if
The scenario called for a lightweight, serverless mechanism to trigger a single recurring webhook or kick off an entire Cloud Composer DAG on a recurring schedule.
Upload a bash script to a Cloud Storage bucket that runs multiple bq command-line queries sequentially in a continuous background loop.
Upload a bash script to a Cloud Storage bucket that runs multiple bq command-line queries sequentially in a continuous background loop.
What this approach involves
This approach involves writing custom shell scripts using the bq CLI tool or BigQuery client libraries, uploading them to Cloud Storage, and attempting to execute them in an ad-hoc virtual machine, container, or background process.
Why it doesn't meet the requirement
- Not a managed Cloud Composer orchestration pattern (✗): Cloud Storage stores DAG definition files, plugins, and logs for Cloud Composer, but it is an object storage service, not a compute execution environment for shell scripts.
- No native monitoring or retry framework (✗): Custom scripts lack the enterprise monitoring, backfill management, visual graph lineage, and centralized logging provided by Cloud Composer.
- Operational burden: Managing execution environments, server availability, authentication tokens, and custom retry logic introduces unnecessary maintenance overhead.
Key characteristics and limitations
- Storage vs. compute: Cloud Storage is purely for storing objects and DAG source files; it does not execute scripts directly.
- High maintenance cost: Custom scripting requires extensive manual error-handling and alerting logic.
- Lack of UI visibility: Does not integrate with the Airflow web server or DAG UI to give data teams visibility into task states and run history.
This answer would be correct if
You were simply archiving migration scripts or manual administration tools into Cloud Storage for backup and administrative reference.
Define a Directed Acyclic Graph (DAG) in Python that models each step as a task and establishes explicit upstream and downstream dependencies between them.
Define a Directed Acyclic Graph (DAG) in Python that models each step as a task and establishes explicit upstream and downstream dependencies between them.
What Cloud Composer and DAGs are
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. In Airflow, workflows are authored as Directed Acyclic Graphs (DAGs) using standard Python code. A DAG defines a collection of all tasks to run, organized in a way that reflects their relationships and dependencies without circular loops.
How it meets the requirements
- Explicit task sequencing (✓): In a DAG, tasks represent discrete units of work (such as executing a BigQuery job via dedicated Airflow operators). You set dependencies explicitly using bitshift operators (e.g.,
task_a >> task_b) or relationship methods. - Eliminating race conditions (✓): Airflow's scheduler evaluates upstream task states. The transformation task will not start until the staging ingestion task succeeds, eliminating race conditions caused by static time estimates.
- Error handling and state management (✓): If a task fails, downstream tasks are automatically marked as
upstream_failedand skipped or retried according to configured retry policies.
DAG Task Chain Prevents Race Conditions
Process Flow
Key advantages of DAG-based orchestration
- Programmatic workflow definitions: Define workflows as code in Python, enabling source control, code reviews, unit testing, and continuous integration.
- Rich ecosystem of Google Cloud operators: Leverage native Airflow operators for BigQuery, Cloud Storage, Dataproc, and other services to manage jobs without writing boilerplate API client code.
- Centralized monitoring and visual DAG UI: Inspect real-time execution graphs, task run duration, historical trends, and logs in the Airflow UI or Cloud Composer DAG UI.
- Flexible scheduling and triggers: Schedule DAGs based on cron expressions, logical dates, dataset events, or manual triggers.
Why this is optimal
Simple time-based schedulers (such as Cloud Scheduler or independent BigQuery scheduled queries) trigger jobs at fixed clock times without knowing whether previous jobs finished or failed. Modeling pipelines as DAGs in Cloud Composer ensures data integrity, handles variable task runtimes gracefully, and provides comprehensive monitoring and alerting.
Try 10 interactive questions like this one, free
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
- 1,132 practice questions across all 4 exam domains
- Every answer explained: why it is right, and why the others are not
- Scout, an AI tutor that answers your follow-up questions
- Readiness tracking by domain, so you know when to book the exam
Related questions
Keep the momentum going with these hand-picked practice scenarios
FundamentalAssociate Data Practitioner
BigQuery Dry Run
Create and manage scheduled queries (e.g., BigQuery, Cloud Scheduler, Cloud Composer)
FundamentalAssociate Data Practitioner
Cloud Scheduler Retry Policy with Exponential Backoff
Create and manage scheduled queries (e.g., BigQuery, Cloud Scheduler, Cloud Composer)
FundamentalAssociate Data Practitioner
Cloud Storage Dual-Region
Distinguish between primary and secondary data storage location type (e.g., regions, dual-regions, multi-regions, zones) for data redundancy
FundamentalAssociate Data Practitioner
Uniform Bucket-Level Access
Compare methods of access control for Cloud Storage (e.g., public or private access, uniform access)
FundamentalAssociate Data Practitioner
Avro vs Parquet Data Formats
Distinguish the format of the data (e.g., CSV, JSON, Apache Parquet, Apache Avro, structured database tables)
AdvancedProfessional Cloud Security Engineer
Regional External Application Load Balancer
Configuring network perimeter controls (e.g., Cloud Next Generation Firewall [Cloud NGFW] rules and policies, Identity-Aware Proxy [IAP], load balancers, and Certificate Authority Service).
Last updated