Prepare and test your skills

Prepare and test your skills

Evaluating a BigQuery (opens in a new tab) job means checking its completion status, execution stages, slot usage, and bytes processed to judge speed, cost, and errors. A BigQuery job moves from submission to distributed execution to persistent result storage, and each phase leaves signals in job history, metrics, and logs. Healthy jobs complete with expected slots and bytes, while slow or failed jobs show queuing, high billed bytes, quota errors, or imbalance between stages.
BigQuery query execution is the distributed process that turns SQL text into parallel work and then stored results. BigQuery first builds an execution tree that breaks the query into stages containing steps that can run in parallel. Stages then exchange intermediate data through a fast distributed shuffle tier that uses a petabit network and RAM to move data to worker nodes, while additional monitoring workers direct progress and can adjust the plan as stage results arrive. When the query finishes, BigQuery writes results to persistent storage and returns them, which lets the next identical run be served from cached results.
A process flow showing a SQL query being parsed into an execution tree of stages, exchanged through the distributed shuffle tier, written to persistent storage, and returned, with monitoring via the query plan.
The query plan is the record of stages and steps that BigQuery generated for a job, plus how each stage performed. The query plan can be viewed in the Google Cloud console in graphical form as the query execution graph, with performance insights for running or completed queries. The same execution details can be requested through INFORMATION_SCHEMA.JOBS views or the jobs.get REST API method. A stage that writes much more output than other stages often means filtering should happen earlier in the query, and because BigQuery can change the plan mid-run, repeated runs on the same data can vary even without a code change.
Slots are BigQuery compute capacity for query jobs, and bytes processed and billed measure how much data a job scanned. Slots are allocated per billing account, and multiple projects can share the same reservation of slots. Key monitoring signals for a job and project include:
BigQuery pricing changes what slot and byte signals mean, so cost review follows the model in use:
| Model | What you pay for | Capacity behavior |
|---|---|---|
| On-demand pricing | Bytes scanned by queries | Fixed query-processing capacity per project |
| Capacity-based pricing | Dedicated capacity purchased through reservations | Projects share reserved slots by assignment |
Choose cost review by bytes for on-demand work and by reservation use for capacity-based work, since the same slow query can be a scanning problem in one model and a slot-contention problem in the other.
Job completion status shows whether a BigQuery job succeeded, is still running or queued, or failed with an error. BigQuery Studio exposes job history on a per-user or per-project basis for quick checks in the Google Cloud console. Cloud Monitoring adds project signals such as queries in flight, number of queries executed, and percentile execution times for non-cached queries. Cloud Logging audit logs record job lifecycle events with filters for query started and jobs completed based on JobService.InsertJob or JobService.Query on resource type bigquery_project. Together, history answers what happened, metrics answer how the system behaved, and logs answer when and why a specific job changed state.
Exam tip: The query execution times metric covers only non-cached queries, so a fast repeat run served from cached results will not appear in those percentiles.
Anomaly detection in BigQuery compares a job's stages, slots, bytes, and status against normal behavior for the same workload. BigQuery runs many queries in parallel and queues queries to run when resources are available, then redistributes resources fairly between new and running queries as queries start and finish. That fair sharing means performance depends on the number of concurrent queries rather than submission order, so a sudden rise in queries in flight or percentile execution times points to contention rather than a single bad query. A large-query signal such as billed bytes above 1,073,741,824 bytes points to a scanning problem, while a quota-exceeded signal with status code 8 and severity WARNING or higher points to a limit problem that custom quotas and billing alerts are designed to control.
Interpreting Dataflow job states and execution metrics means checking the overall job status and then reading per-step status and performance numbers to judge health. A Dataflow job is monitored through the Dataflow monitoring interface, which provides a job summary, a job log, and a graphical representation called the job graph (opens in a new tab). Data flows from one transform to the next through the graph, and each step reports its own status and timing so slow or failing parts can be found quickly.
Locating a Dataflow job starts from the Dataflow Jobs list in the Google Cloud console. Job names include the pipeline name plus a unique ID attached with a hyphen, which distinguishes each run of the same pipeline. The Job info panel shows the overall status of the selected job, and the Jobs list also shows status for every job.
To open the detailed view for a job:
When a batch job completes successfully, the Job status field updates to Succeeded. If the job fails or has errors, the next step is to check the job logs or worker logs for details before retrying or fixing the pipeline.
The Job graph represents each transform in the pipeline as a box connected in execution order. By default the Job graph page displays Graph view, which shows the pipeline shape visually from source transforms through intermediate steps to the destination. Table view contains the same step information in a different format and supports sorting.
| View | What it shows | Choose when |
|---|---|---|
| Graph view | Each transform as a box with status and metrics | The pipeline is small enough to follow visually and you want to trace flow between steps |
| Table view | Same job steps as rows in a table | The job has many stages that are hard to navigate, or you want to sort steps by a property such as wall time to find slow steps |
If a step represents a composite transform, the box can be expanded to view its sub-transforms. To expand the step, click the Expand node arrow, which reveals the nested transforms that execute inside that parent step.
Step status and execution metrics in the Job graph show whether each part of the pipeline is progressing, finished, or blocked. Each box contains the transform name, its current status, and key measurements including data lag, wall time, maximum operation latency, and the number of job stages that execute the step. Wall time is especially useful for finding slow steps, while data lag and operation latency help spot steps where data is waiting or operations are slow.
The status in each box is one of the following:
A step with Succeeded means that portion of the pipeline finished its work, while Failed or Unknown means that step needs investigation. Because the graph preserves order, a failure in an early transform prevents downstream transforms from receiving data, so health is judged step by step from source to sink.
Overall job health in Dataflow is determined by combining step status, detailed step information, logs, and bottleneck signals. When you click a step in the job graph, the Step Info panel shows more details about that step, including its timing and bottleneck cause. If Dataflow detects a bottleneck, the job graph shows a warning alert symbol on the affected steps, and opening the Step Info panel reveals the cause. Transform names in the graph come either from a name assigned by the developer or from a name inferred from the code, such as a class name or DoFn function name.
Exam tip: Do not use personally identifiable information in transform names, because transform names appear in the monitoring interface, log files, and debugging tools.
After the Job status shows Succeeded, health can be confirmed by querying the destination, such as running a standard SQL SELECT query against the output BigQuery table to verify that rows were added correctly.
Diagnosing errors and troubleshooting job failures is the practice of using job status, execution details, and logs to find why a Dataflow (opens in a new tab) or BigQuery (opens in a new tab) job failed or stalled and what to fix. Dataflow jobs are diagnosed through the Dataflow monitoring interface and worker logs, while BigQuery jobs are diagnosed through the query plan, execution graph, and job metadata. Cloud Logging filters connect both services to detailed error and warning records.
Dataflow job diagnostics are the status signals that show whether a unified stream and batch pipeline is progressing, stalled, or failed. Dataflow is a fully managed service where data flows from one or more sources through transforms to a destination, and the Dataflow service allocates a pool of worker VMs to execute that pipeline. When the job completes or is cancelled, Dataflow automatically deletes the VMs, so a job that never finishes but keeps workers running points to stalled processing rather than normal completion.
The Dataflow monitoring interface in the Google Cloud console provides a graphical representation of the pipeline, showing the progress and execution details of each pipeline stage. The job details view breaks that picture into three checks:
Worker logs flow from the worker VMs to Cloud Logging, where a filter on resource.type="dataflow_step" and log_id("dataflow.googleapis.com/worker") with severity>=WARNING isolates errors and warnings from Dataflow workers. Because Dataflow can autoscale and dynamically rebalance work among VMs, a failure that affects only one stage while other stages continue often points to code, data, or destination access for that stage rather than a lack of workers.
BigQuery job diagnostics are the query execution records that show how a query job moved through planning, execution, and result writing. BigQuery runs query jobs submitted through the Google Cloud console, the bq query command, the jobs.query or jobs.insert REST API methods, or the client libraries. Several processes occur when BigQuery runs a query, and each process leaves details that help isolate a failure:
The query plan can be viewed in the Google Cloud console to troubleshoot or optimize performance, and execution details can be requested with the INFORMATION_SCHEMA.JOBS views or the jobs.get REST API method. Because BigQuery runs many queries in parallel, can queue queries to run when resources are available, and fairly redistributes resources between new and running queries, a stalled job may be queued behind concurrent work while a failed job will carry an error status and failed stage in those execution details.
Cloud Logging filters are precise queries that isolate Dataflow worker problems and BigQuery job problems by resource type, log name, and status. The filters use fields such as resource.type, log_id, protoPayload.methodName, protoPayload.status.code, and severity to separate normal completion messages from warnings and errors. Running the narrow filter first prevents a failed stage from being hidden among routine audit records.
| Check | What it isolates | Filter pattern |
|---|---|---|
| Dataflow worker errors and warnings | Errors from pipeline workers | resource.type="dataflow_step" with log_id("dataflow.googleapis.com/worker") and severity>=WARNING |
| BigQuery jobs completed | Finished query and insert jobs | resource.type="bigquery_project" with log_id("cloudaudit.googleapis.com/data_access") and methodName of JobService.InsertJob or JobService.Query |
| BigQuery quota exceeded | Jobs blocked by quota | resource.type of bigquery_dataset or bigquery_project with protoPayload.status.code=8 and severity>=WARNING |
| BigQuery large queries | High-cost queries over 1073741824 billed bytes | resource.type="bigquery_project" with totalBilledBytes > 1073741824 |
A Dataflow failure usually traces from the failed stage in the monitoring interface to the matching worker error record, which then points to a corrective action such as fixing pipeline code, source access, or sink permissions. A BigQuery failure usually traces from the failed job record to the query plan stage, quota status, or billed-bytes size, which then points to rewriting the query, waiting for queued resources, or requesting more slots or quota.
Exam tip: A BigQuery quota failure is identified by protoPayload.status.code=8, while a large-query check uses totalBilledBytes > 1073741824.
Succeeded in the Job status field on success, and Dataflow step status is Running, Queued, Succeeded, Stopped, Unknown, or Failed.resource.type="dataflow_step" and log_id("dataflow.googleapis.com/worker") with severity>=WARNING.protoPayload.status.code=8, and large queries are identified by totalBilledBytes > 1073741824.Check BigQuery Studio job history on a per-user or per-project basis for quick status checks in the Google Cloud console, supplemented by Cloud Monitoring for queries in flight and percentile execution times and Cloud Logging audit logs for job lifecycle events. Healthy jobs complete with expected slots and bytes, while slow or failed jobs show queuing, high billed bytes, quota errors, or imbalance between stages.
Review bytes scanned for on-demand pricing work and reservation use for capacity-based pricing work. On-demand pricing charges for bytes scanned by queries with fixed query-processing capacity per project, while capacity-based pricing charges for dedicated capacity purchased through reservations that projects share by assignment, so the same slow query can be a scanning problem in one model and a slot-contention problem in the other.
Go to Dataflow Jobs in the Google Cloud console, click the job name to open its details, and open the Job graph tab to see the execution graph, summary, and logs. Each transform shows status as Running, Queued, Succeeded, Stopped, Unknown, or Failed with wall time, data lag, and operation latency, and Table view supports sorting by properties such as wall time while the Step Info panel reveals bottleneck causes.
An administrator is monitoring an active Cloud Dataflow streaming pipeline that uses Streaming Engine. During a sudden surge in incoming traffic, message latency increases and an unprocessed backlog accumulates, but Dataflow stops provisioning additional worker instances.
Which chart in the Dataflow monitoring interface should the administrator inspect to determine the specific reason the autoscaler did not provision more workers?
The Autoscaling rationale chart
The Execution graph step throughput view
The Worker CPU utilization chart
The Backlog metric chart
Last updated