Prepare and test your skills
Prepare and test your skills
Creating and managing scheduled queries means running BigQuery (opens in a new tab) SQL automatically on a timetable and saving the results to a table. A scheduled query is managed by the BigQuery Data Transfer Service as a transfer configuration that stores the query text, schedule, destination table, and credentials. Cloud Composer (opens in a new tab) is used when the work grows beyond a single timed query into a multi-step workflow with dependencies.
BigQuery scheduled queries automate repeated SQL without manual runs or custom pipeline code. A scheduled query can run a GoogleSQL SELECT, DDL, or DML statement on a timetable, and a SELECT query can write its results to a destination table. The same mechanism can copy Bigtable data accessed through an external table into BigQuery on a recurring basis. This makes scheduled queries useful for daily rollups, recurring imports, and pre-computed results.
The transfer configuration controls every run in the same way. It holds the query text, the repeat schedule, the destination dataset and table, the data location, and the identity used to run it. When the scheduled time arrives, the service runs the query with those stored credentials and applies the write preference of Append to table or Overwrite table. Because destination, location, and identity are stored with the schedule, downstream tables stay up to date.
Periodic execution turns a tested query into a recurring transfer that runs on its schedule and writes to the same destination each time. Setup follows a fixed order: first test the SQL in BigQuery and confirm correct results, then create the schedule with a display name and repeat interval, then set the destination table and write preference, then set location and service account identity, then enable monitoring and save. Data flows from the scheduled query execution to the configured destination table on every run.
Schedules use Cron-like values with a minimum interval of 5 minutes between runs. Common patterns are every mon 23:30 for weekly, every 6 hours for four times per day, and every hour on mon,tue,wed,thu,fri for weekdays only. An optional start time can delay the first run and an On-demand option saves the query with no schedule for manual runs. If the start time is later than the next scheduled time, the first run waits until the following cycle.
Failure handling combines automatic retries with alerts so repeated failures are not silent. Temporary errors such as rateLimitExceeded or unavailable are retried with exponential backoff and jitter, where waits grow progressively longer, and official client libraries handle many gRPC retries automatically. Owners can enable email notifications for transfer-run failures and set a Pub/Sub topic to receive run status. Failed load jobs still count toward quotas, so fewer, larger writes spread across the day help periodic jobs stay within limits.
Exam tip: The minimum duration between BigQuery scheduled query runs is 5 minutes, and On-demand saves the query with no schedule for later manual runs.
Cloud Composer, also called Managed Service for Apache Airflow, is a fully managed workflow orchestration service for extract, transform, load (ETL) and extract, load, transform (ELT) workloads across systems. Unlike simple time-based scheduling, Cloud Composer manages task order, dependencies, and retries for durable production and MLOps workflows. A downstream task such as a BigQuery job runs only after the upstream tasks it depends on have finished.
Work is defined in a Directed Acyclic Graph (DAG), which is a workflow definition that ensures each task runs at the right time and in the right order. Each task is a single unit of work, such as preparing data for ingestion, monitoring an API, sending an email, or running a pipeline. A DAG can start on a schedule, manually, or in response to an event such as a change in a Cloud Storage bucket. This allows BigQuery jobs to run as part of larger time-driven and event-driven workflows rather than only at fixed times.
Each managed Airflow environment is a self-contained deployment based on Google Kubernetes Engine (GKE) that connects to other Google Cloud services through built-in connectors. The environment includes a GKE cluster that runs schedulers, triggerers, workers, Composer Agent, and monitoring, an Airflow web server that hosts the Airflow UI, an Airflow database that holds metadata about DAGs and runs, and a Cloud Storage bucket that stores DAG files, logs, plugins, and environment data. Data flows from the bucket to the GKE workers when DAG files are read, and results flow back as logs to Cloud Logging and metrics to Cloud Monitoring. Cloud Composer also provides over 1,000 prebuilt operators, Airflow connections, custom plugins, Python dependencies, and a declarative YAML framework for building pipelines.
Optimization for scheduled queries means scanning less data, writing efficiently, and avoiding quota problems. Partitioned tables divide a large table by date, timestamp, or integer so each run reads or writes only the needed partition. A common pattern collects the day's data in a staging table and loads it once into that day's partition, such as the partition for 2025-09-18. Avoid partitioning when partitions would stay consistently smaller than 10 GB.
Other practices reduce cost and speed recurring work:
INFORMATION_SCHEMA.JOBS views, or the jobs.get API to find stages with large output and add filters earlier.Monitoring Dataflow (opens in a new tab) in the Google Cloud console means tracking a job from pipeline definition to running execution to finished result. A pipeline defines how data flows from sources through transforms to sinks, and a job is the running execution of that Apache Beam pipeline on managed workers. The Dataflow job interface shows job status, stage graphs, and performance charts to verify success and locate bottlenecks, delays, or failures.
Job status is the operational state reported by the Dataflow service for each job in the jobs list (opens in a new tab) at Dataflow > Jobs. Status is the first health signal because it separates normal progress from jobs that need attention. Preparation states come first, active processing comes next, and terminal states come last.
The lifecycle moves in order from creation through active work to a final result. Starting and Queued come before Running, while Canceling, Draining, and Updating are transitions that end in Canceled, Drained, or Updated. Succeeded means batch work finished normally, while Failed means the job did not complete and needs investigation. A status of — means the monitoring interface has not yet received a status from the service.
Common statuses include Starting when the system is preparing to launch, Queued for FlexRS or Flex Template launches, Running for active work, and Draining where in-flight data finishes while new input stops. Drained, Canceled, Updated, and Succeeded are final states that stop active processing. Failed is the final state that triggers troubleshooting in logs and stage details.
The Job details page is the per-job monitoring view that opens when a job name is selected in the Dataflow monitoring interface. The page combines a visual pipeline graph with tabs for performance, cost, and scaling, plus logs and diagnostics for troubleshooting. Switching between Job graph, Execution details, Job metrics, Cost, and Autoscaling changes the lens from structure to behavior to resource use.
The supporting panels explain why a stage is slow or failing. Job logs show service-level messages for the whole job, Worker logs show step-level messages from Compute Engine instances for the selected graph step, and Diagnostics shows where errors clustered on a timeline with recommendations. Selecting a step in the graph shifts the logs panel from broad orchestration to narrow code execution. Job info, data sampling, and autoscaling views add description, data observation, and streaming scaling events.
Stage-level details break progress down by workers and work items within each stage. A stage corresponds to a fused step or group of transforms, and work items are the units of data assigned to workers in that stage. When overall metrics show slow progress, execution details help isolate which stage holds backlogged work or underused workers. This link from job-level progress to stage-level workers moves troubleshooting from noticing a problem to locating its source.
A Dataflow pipeline stage is one processing step in a sequence that reads data, transforms or aggregates it, and writes results to a destination. Typical work includes moving data as-is, making data more usable for the target, aggregating and enriching for analysis, and joining with other data. The Apache Beam pipeline defines these steps, and the Dataflow runner executes them by uploading code to a Cloud Storage bucket and allocating virtual machines.
A typical streaming extract, transform, and load flow moves through connected systems in this sequence:
Data flows from Pub/Sub to Dataflow to BigQuery to Looker, so a delay in Dataflow propagates downstream as late warehouse data and stale dashboards. The monitoring interface highlights stage progress and execution details to reveal whether latency comes from reading, transforming, or writing.
The bottleneck detector (opens in a new tab) is the streaming-job monitor that highlights the component that limits throughput and causes backlogs. Streaming steps connect through queues, so when a downstream step is slower than upstream production, the queue fills and backlog propagation slows upstream steps. The detector watches shuffles, DoFn threads, and state commits, and when a delay exceeds five minutes it marks the step in the execution graph with diagnostic information.
An active bottleneck falls into one state that describes whether intervention is needed: processing is stuck with no progress, processing is ongoing but falling behind as backlog grows, processing is ongoing with steady backlog at the incoming rate, or processing is ongoing and catching up from a backlog. Catching up at startup with existing backlog is normal and does not require intervention. For recurring batch jobs, health is judged by job duration versus objective in Thread time per step graphs, while streaming health is judged by data freshness versus guarantee plus CPU and memory use.
Troubleshooting follows an ordered flow from signal to location to action. First check job status in the jobs list, then open Job details to find the slow stage, then filter logs and open Diagnostics.
| Location | Source | Use when |
|---|---|---|
| Job logs | Dataflow service for the whole job | Validation, permissions, or resource access fails before one step stands out |
| Worker logs | Compute Engine workers for the selected step | A specific transform shows an exception such as a buggy DoFn in ParDo |
| Diagnostics | Aggregated errors on a timeline | Errors spike and the affected step and recommendation must be isolated first |
To isolate errors, first show Job logs, then filter severity to Error, then expand the entry for stack trace or permission detail. Graph construction errors create no job and appear only where the pipeline ran, while validation, worker-code, and transient service errors appear as a Failed job with logs and diagnostics.
Exam tip: A streaming delay over five minutes registers as a bottleneck, but catching up from a backlog at startup is normal and does not require intervention.
Continue with this study guide: Monitor Dataflow pipeline progress using the Dataflow job UI
Cloud Logging (opens in a new tab) is a fully managed service that stores, searches, analyzes, monitors, and alerts on log data and events. Log data flows from Google Cloud services and Amazon Web Services into Cloud Logging, while data from more than 80 additional applications and hybrid systems flows in through Bindplane. Cloud Monitoring (opens in a new tab) is the integrated service that turns performance information and log-based metrics into charts, dashboards, and alerts. Together they let a data team trace a pipeline failure from symptom back to cause and watch reliability over time.
Log categories in Cloud Logging describe who wrote each entry and what question it answers, and the categories are not mutually exclusive. Platform log entries are written by Google Cloud services for debugging service behavior. Component log entries come from Google-provided software on customer systems, such as GKE components, for support. Security log entries answer who did what, where, and when. User-written entries come from custom applications through the Ops Agent, API, or client libraries. Multicloud and hybrid entries bring in activity from Microsoft Azure and on-premises infrastructure.
Three log types are especially useful for container troubleshooting. Node and runtime logs from kubelet and containerd record Pod lifecycle, image pulls, Out of Memory events, probe failures, volume errors, and node NotReady status. App logs from stdout and stderr record crashes and errors inside containers. Audit logs record control-plane API calls, such as who deleted a Deployment. If app logs are empty, node and audit logs often reveal scheduling, resource, or image-pull problems.
Logs Explorer is the Cloud Logging user interface for querying, viewing, and analyzing log entries. A user selects a resource scope, writes a query in the Logging query language, runs the query, and opens an entry to see its full message, JSON payload, metadata, and timestamp. Filtering first by resource, severity, and time narrows millions of entries to the set related to a failing pipeline step.
Isolation follows a fixed order:
Common filters select the monitored resource type such as k8s_container, the log stream such as stdout or stderr, the resource labels such as pod name and namespace, the severity such as ERROR or CRITICAL, and text inside jsonPayload.message. For example, error logs for one container can be isolated with resource type plus pod name plus severity.
Exam tip: Use severity ERROR or CRITICAL to find failures, and use audit logs to find who changed configuration when app logs show no code error.
Log storage in Cloud Logging starts automatically in the resource where the data originates, such as Compute Engine logs stored in the same project. Entries are kept for a defined retention period and then deleted, with configuration controlling which logs are kept, discarded, and where they are stored. Access to log buckets is controlled by Identity and Access Management (IAM) permissions and roles.
The Log Router is the component that forwards copies of entries from their source to a destination, including destinations in a different resource. Supported destinations are a Google Cloud project, a log bucket, a BigQuery dataset for security and access analytics, a Cloud Storage bucket for archival storage, and a Pub/Sub topic that supports third-party integrations like Splunk or Datadog. This routing lets teams keep operational logs in place while sending security or long-term copies elsewhere.
Log-based metrics turn specific log entries in Cloud Logging into quantifiable time-series data in Cloud Monitoring. The metric extracts data from each matching entry and sends the time series from Cloud Logging to Cloud Monitoring for charting and alerting. This translation connects raw log text to measurable signals for pipeline health.
Log-based metrics support three main uses: monitoring the count of a message such as error-status entries, observing trends in numeric data such as latency with notification on unacceptable change, and creating charts of extracted numeric data. Visualization starts with Google-managed dashboards created automatically for resources, and teams create custom dashboards when they need to control what metric data is viewed and how it is displayed. A failure or latency metric then appears as a line over time that reveals climbing errors or drifting latency.
An alerting policy in Cloud Monitoring is the rule that watches a metric and sends notifications when defined criteria are met. The policy contains the condition, such as error count over a threshold, plus the notification channels. Supported channels include email, Cloud Mobile App, PagerDuty, and Slack. When the condition is met, Cloud Monitoring notifies each channel and stores an alert as a persistent record with status, log links, metric chart, labels, and duration. Policies can be managed in the console, API, CLI, or Terraform.
Continue with this study guide: Review and analyze logs in Cloud Logging and Cloud Monitoring
Selecting a data orchestration solution means matching how work is defined, triggered, and connected to business needs for timing, location, control, and management effort. A managed open-source environment runs familiar orchestration code with full scheduling control, while serverless execution runs ordered steps across APIs without managing infrastructure. The correct choice depends on whether the workload needs complex data pipelines, lightweight service coordination, or job-specific automation.
Cloud Composer is Google Cloud's fully managed workflow orchestration service built on Apache Airflow. Teams use Cloud Composer to author, schedule, and monitor pipelines that span cloud environments and on-premises data centers from one managed control plane. The service depends on directed acyclic graphs (DAGs) to define scheduling and task order, and DAGs depend on operators to run specific technologies.
In Cloud Composer, data flows from one operator task to the next inside a DAG in the order the DAG defines. Operators provide ready-made actions for multi-cloud technologies, including extract and load steps, transformations for ELT, and REST API calls. For example, a DAG can replace a Hive operator with a BigQuery operator so the same ETL pattern loads and transforms data in BigQuery. Cloud Composer fits when pipelines need open-source Airflow logic, rich scheduling, and control across hybrid and multi-cloud systems.
Workflows (opens in a new tab) is a fully managed, serverless service for sequencing loosely coupled services in a defined order. A workflow separates coordination logic from core business logic, so dependencies between services stay explicit and observable end to end. Workflows can combine any HTTP-based API and can wait for all operations to complete, and execution can be event-driven.
A workflow is defined as steps written in YAML or JSON using the Workflows syntax. The definition depends on standard library functions, environment variables, and lightweight serverless connectors to build arguments, process responses, and call Google Cloud APIs. Connectors handle request formatting and expose methods and arguments, so the workflow avoids low-level API details. A typical event-driven sequence starts when a Cloud Run function triggers Workflows after a Cloud Storage upload, then data flows from Workflows to language and vision APIs for recognition, then tags flow to Firestore for storage. Workflows fits lightweight coordination of existing HTTP APIs in a defined order with less code.
Choosing between orchestration models means matching how much infrastructure is managed and how integration is expressed. Managed open-source orchestration fits durable, scheduled data pipelines that reuse Airflow patterns across environments. Serverless orchestration fits coordination of existing services where triggers, connectors, and ordered steps matter more than cluster management.
| Service | Definition model | Execution and integration model |
|---|---|---|
| Cloud Composer | DAGs written with Airflow code and operators | Managed Airflow environment that schedules and monitors tasks across cloud and on-premises |
| Workflows | Steps in YAML or JSON with standard functions | Serverless execution that calls HTTP APIs and connectors in order, often event-driven |
| Dataproc Workflow Templates | Managed template for an ETL pipeline | Runs with Managed Service for Apache Spark to manage Spark-based execution |
Timing requirements describe how often a workload runs and whether its schedule can shift. Long-running applications should run in the region with the highest carbon-free energy (CFE) percentage because they operate continuously and accumulate impact. Batch workloads have greater flexibility because start times can be predicted and shifted. This flexibility enables time-shifting, where non-urgent work such as model training runs during off-peak or high solar and wind hours without affecting performance or cost.
Location requirements define where data and compute may run and how quickly users must be reached. Data residency and sovereignty dictate storage location for compliance, latency depends on distance between users and deployment region, and cost varies because pricing differs across regions. A region strategy combines data-driven selection using CFE metrics and Low CO2 indicators, policy-based governance using organization-policy location constraints, and operational flexibility using time-shifting and carbon-aware scheduling.
Exam tip: Choose the highest-CFE region for a long-running workload, but use time-of-day scheduling for a flexible batch workload that can run during higher-CFE hours.
An automation approach defines who provisions infrastructure and who installs workload software. The lifecycle order is fixed: first provision infrastructure such as Compute Engine instances and disks, then install systems after infrastructure is ready. Terraform provides declarative files for repeatable provisioning, Workload Manager Guided Deployment Automation provides guided steps plus exportable Terraform and Ansible files, manual deployment through console, CLI, and APIs provides complete control, and Marketplace or appliance libraries provide partner or trial deployments.
Cluster options form a spectrum from managed lifecycle to self-managed control. Cluster Director provides a Google-managed Slurm lifecycle for researchers, Google Kubernetes Engine (GKE) provides container orchestration and scaling for platform and ML engineers, Cluster Toolkit provides validated blueprints for team-managed Slurm or Kubernetes clusters, and Compute Engine with custom or no orchestrator provides granular OS and networking control for architects. Lower complexity means Google handles more lifecycle work, while higher complexity means the team handles setup and maintenance in exchange for control.
The decision flow asks three questions in order: whether cluster orchestration is needed, whether standard containerized apps will run, and whether Google should manage the lifecycle. No orchestration need points to Compute Engine, containerized need points to GKE, managed lifecycle need points to Cluster Director, and team-managed need points to Cluster Toolkit. After the flow, the team verifies that management level and features meet workload and skill requirements.
Event-driven ingestion from Pub/Sub (opens in a new tab) to BigQuery (opens in a new tab) means producers emit events as they happen and BigQuery stores them for near-immediate SQL analysis. Pub/Sub is an asynchronous and scalable messaging service that decouples publishers from subscribers, and BigQuery is the analytics warehouse where events land for queries and dashboards. Real-time ingestion keeps data continuously fresh, while batch processing collects data on a schedule for complete and validated results.
How Pub/Sub decouples publishers and subscribers is through asynchronous messaging with shared topics and independent subscriptions. Publishers send events to the Pub/Sub service without knowing how or when they will be processed, and subscribers receive events from subscriptions independently. Typical Pub/Sub latency is on the order of 100 milliseconds. This model increases flexibility and robustness compared to synchronous calls where a slow subscriber slows the publisher.
Data flows from publishers to a shared topic and then from the topic to each subscriber, so publishers and subscribers scale and fail independently. The subscription also buffers data if downstream processing stops, and backed-up events are processed when the pipeline runs again. The same raw or processed event can feed BigQuery, Bigtable, Cloud Storage, and other applications at the same time. This many-to-many distribution supports real-time distribution, database change replication, parallel task distribution, and an enterprise event bus.
The ordered pipeline from Pub/Sub to BigQuery moves data from event sources to analytics storage through dependent stages. Source systems publish user interactions, server events, or sensor readings to a Pub/Sub topic. Pub/Sub holds messages and delivers them to subscriber clients, Dataflow processes them, and BigQuery stores them for analysis. Each stage depends on the prior stage for delivery of buffered events and transformed rows.
Data flows in a fixed order:
Real-time ingestion is required when decisions, alerts, or experiences depend on data seconds old rather than hours old. Traditional batch processing collects data and loads it on a schedule, which delays visibility until the job runs. Real-time fits streaming analytics on clicks and transactions, logging from many clients at once, monitoring across teams as an event bus, and Internet of Things (IoT) streaming from sensors into processing pipelines.
Batch processing is the scheduled collection and transformation of accumulated data to meet a completeness or deadline requirement. A common example is a business-critical daily report that must be completed by a fixed deadline for a regulator, where running duplicate pipelines in different regions can be worth the cost. Batch ingestion often combines streaming capture with scheduled transformation, where raw events land in BigQuery raw tables and Workflows triggers an hourly Dataflow pipeline that transforms them into curated tables and views. Cloud Storage provides temporary processing space and archiving for failed events.
| Option | Data freshness | Choose when |
|---|---|---|
| Real-time ingestion | Continuous, near-immediate | Results must reflect current activity such as live dashboards and trend detection |
| Batch processing | Scheduled, such as hourly or daily | Results must be complete and validated by a deadline such as a regulator report |
BigQuery subscriptions are a type of Pub/Sub subscription that writes messages from a topic directly into a BigQuery table without an intermediate service. Publishers send events without waiting for BigQuery, and the subscription forwards messages directly to the configured table where rows become available for queries and visualization. The subscription depends on the topic for collection and the destination table for storage, and message fields must align with the table schema.
The choice between direct delivery and Dataflow depends on whether events need processing before storage. A BigQuery subscription is simplest when low latency and low operational overhead matter most and little or no transformation is needed. A Dataflow pipeline is required when events need cleaning, enrichment, SQL joins, filtering, or delivery to multiple systems. Direct streaming fits IoT telemetry, application and server logging, user interaction events, and database change events where many sources generate continuous events for a single fresh dataset.
Exam tip: Use a BigQuery subscription for direct Pub/Sub to BigQuery streaming, and use Dataflow when the flow requires processing before BigQuery.
Eventarc (opens in a new tab) is a Google Cloud event-routing service that carries events from a source to a destination. Eventarc Standard is the routing layer, and an Eventarc trigger is the rule that listens for a specific event and starts work when it occurs. This model improves efficiency because processing runs only when something happens instead of running on a fixed schedule or polling. For example, a new Cloud Storage object can start an integration flow and a Pub/Sub message can call a container service.
An Eventarc trigger connects an event source to an event destination through matching rules and identity. The source produces the event, Eventarc routes the event when trigger filters match, and the destination receives the event as a request to start work. A common flow is Cloud Storage to Eventarc to Application Integration, or Pub/Sub to Eventarc to Cloud Run (opens in a new tab). For Google Kubernetes Engine (GKE) destinations, Eventarc creates an event forwarder that pulls events from Pub/Sub and forwards them to the target service.
Trigger type selects the category of source, event provider selects the specific product, and event type selects the exact action such as google.cloud.pubsub.topic.v1.messagePublished. Optional filters narrow the match to a specific bucket, topic, or resource name using exact match without wildcards or regular expressions. Event data can use application/json or application/protobuf content types, where JSON is larger than Protobuf, which matters when a destination limits event size. In data pipelines, the same routing pattern can start Dataform transformation, Dataflow stream processing, Cloud Functions tasks, Cloud Run logic, or Cloud Composer orchestration.
Trigger creation collects the same core configuration whether it starts from Application Integration or from Eventarc, and both paths require activation. Before configuring a trigger, the Eventarc API and Pub/Sub API must be enabled, which requires the serviceusage.services.enable permission through the Owner or Service Usage Admin role. Cloud Run Pub/Sub scenarios also use Artifact Registry, Cloud Build, Cloud Run Admin, and Cloud Logging APIs.
Creation follows an ordered sequence:
Saving the trigger only stores the rules in the trigger definition. The trigger stays inactive until the integration is published or the trigger creation is completed, so publishing is what deploys the rules and starts delivery. It can take up to two minutes after activation to become fully functional. After creation, event source filters cannot be edited, so a change requires creating a new trigger and deleting the old one.
The trigger service account is the Identity and Access Management (IAM) identity that Eventarc uses to call the destination. When an event matches, Eventarc assumes this account and sends the event to the destination URL, and the destination checks the account against its IAM policy before running code. This creates a trust boundary between event routing and compute, so no event can invoke Cloud Run or Cloud Functions without an authorized identity.
If no account is selected, the trigger falls back to the Compute Engine default service account for testing, while production should use a dedicated user-managed account with least privilege. Required roles include Cloud Run Invoker to send authenticated invocations to Cloud Run, Pub/Sub Subscriber to pull events when Pub/Sub is the transport or source, Service Account Token Creator for authenticated push delivery, and Service Usage Admin to enable required APIs. Access can be granted on a single service for tighter control or on the whole project for testing. A network attachment can also connect a producer Virtual Private Cloud (VPC) to a consumer VPC so Eventarc can reach an internal HTTP endpoint.
Reliability in Eventarc depends on region placement, retry behavior, and the activation lifecycle. For best performance and lower latency, the Eventarc trigger, the monitored resource, and the destination service should be in the same Google Cloud region. The Retry on failure option controls redelivery: when enabled, Eventarc retries after a destination failure, and when cleared, Eventarc makes a single attempt with no retries. Retries protect against transient errors, while a single attempt avoids duplicates when the destination cannot handle redelivery.
The lifecycle moves from draft to active only after publishing. First the configuration is saved and remains inactive, then the trigger is connected to the flow or destination, then publishing deploys the rule and moves it to active, then republishing with a changed configuration replaces the old trigger while republishing unchanged reuses the existing trigger. If trigger creation fails during publication, the integration still publishes but the trigger stays inactive until corrected and republished. Publishing with an unchanged trigger reuses the existing trigger.
Exam tip: Saving a trigger only stores its rules — the integration must be published before the trigger becomes active and starts processing events.
Use BigQuery scheduled queries for recurring SQL work that writes results to the same destination table, and use Cloud Composer for multi-step workflows with dependencies. A scheduled query is managed by the BigQuery Data Transfer Service as a transfer configuration that stores the query text, schedule, destination table, and credentials and applies Append to table or Overwrite table, while Cloud Composer uses a Directed Acyclic Graph (DAG) where a downstream task such as a BigQuery job runs only after the upstream tasks it depends on have finished.
BigQuery scheduled queries run on Cron-like schedules with a minimum interval of 5 minutes between runs. Common patterns include every mon 23:30 for weekly runs, every 6 hours for four times per day, and every hour on weekdays only, with an optional start time that delays the first run until the following cycle if the start time is later than the next scheduled time. The On-demand option saves the query with no schedule for later manual runs.
Monitor Dataflow pipeline progress by checking job status in the jobs list and then opening the Job details page for the slow or failed job. Status moves from Starting or Queued to Running and then to Succeeded, Failed, Canceled, Drained, or Updated, and streaming delays over five minutes are marked as bottlenecks in the execution graph. Use Job logs for whole-job service messages, Worker logs for step-level messages from the selected graph step, and Diagnostics for aggregated errors on a timeline with recommendations.
Use a BigQuery subscription when low latency and low operational overhead matter most and little or no transformation is needed, and use Dataflow when events need processing before storage. A BigQuery subscription writes Pub/Sub messages directly into a BigQuery table without an intermediate service, while a streaming Dataflow pipeline reads from the subscription, cleans or enriches events, and writes processed rows using the Storage Write API (gRPC). Dataflow is required for cleaning, enrichment, SQL joins, filtering, or delivery to multiple systems.
Last updated