Analyzing Cloud Trace Waterfalls and Spans

Correlating traces with system metrics and logs is the practice of placing a slow span alongside related measurements and events to explain why latency occurred. Cloud Trace (opens in a new tab) shows where time was spent across services, Cloud Monitoring shows resource and request behavior over time, and Cloud Logging shows discrete events from the same request. Together the three signals let an operator decide whether a bottleneck comes from application code or from constrained infrastructure.

1.1What Metrics, Logs, and Traces Each Contribute

Metrics, logs, and traces answer different questions about the same workload, and correlation depends on using all three together. Metrics are numerical measurements collected at intervals, logs are time-stamped records of discrete events, and a trace represents the journey of a single request through components such as microservices. Default service metrics help reveal component dependencies and their impact on overall performance, while custom metrics published with the Google Cloud SDK extend monitoring to application-specific behavior.

• Metrics: request rate, error rate, response time, CPU consumption, network I/O, disk I/O, and business measures such as orders placed; request rate and error rate can serve as service-level indicators (SLIs).
• Logs: failures, errors, state changes, and application output; logs can include metrics and can be used for SLIs, with examples including Cloud Storage access logs and VPC Flow Logs.
• Traces: the components used in a request journey, where bottlenecks exist, and how long each span and the full journey took.

Active Assist can then analyze utilization trends from metrics to recommend resource adjustments for under-provisioned or over-provisioned components.

1.2Moving Between Trace Waterfalls and Log Entries

The Trace explorer to Logs Explorer pivot connects a slow span to the events emitted by the same request. Trace data flows from instrumented applications through the Telemetry (OTLP) API into a Google Cloud project, where the Trace explorer displays it as a table of traces and a Gantt chart of spans inside the Trace details panel. The Logs and Events tab then links the selected span to the log entries that carry the same trace context.

To move from a trace to its logs, follow this order:

1. Select the trace: In Trace explorer, select the table row for the request of interest, such as the span name /multi.
2. Inspect the span: In the Gantt chart on the Trace details panel, select the span labeled /multi to open request details including method, status code, number of bytes, and caller user agent.
3. Open associated logs: Select the Logs and Events tab to see individual log entries for that trace, expand an entry for details, or select View Logs to continue in Logs Explorer.

The reverse pivot starts from Logs Explorer and returns to the same waterfall. To move from a log to its trace, expand the log entry from the instrumented app, such as handle /multi request, select Traces on a log entry that carries trace context, and then select View trace details to open the Trace details panel for that trace.

1.3Layering Cloud Monitoring Metrics onto Trace Findings

Cloud Monitoring metrics provide the resource and traffic context around the time window of a slow trace. Metrics Explorer queries a selected metric, filters time series to remove unwanted data, and configures aggregation and alignment before display. When measurements are cumulative, Metrics Explorer normalizes them by the alignment period so the chart displays a rate, and when values are integers or doubles, it sums time series by default.

For HTTP services instrumented with OpenTelemetry and Managed Service for Prometheus, the documented query selects http_server metrics under the Prometheus Target resource type and Http category, then sets Aggregation to None to separate routes such as /multi and /single instead of summing them. That separation matters because a trace waterfall may implicate one route while aggregate request rate or error rate hides the difference. Broader resource signals come from monitoring CPU consumption, network I/O metrics, and disk I/O metrics across Google Kubernetes Engine (GKE) (opens in a new tab), Compute Engine, and Managed Service for Apache Spark to detect under-provisioned and over-provisioned resources.

1.4Distinguishing Code Inefficiency from Resource Constraint

Distinguishing code inefficiency from resource constraint requires comparing span structure against infrastructure saturation during the same interval. If CPU, memory, High Bandwidth Memory (HBM) bandwidth, and network utilization remain flat while one span dominates the waterfall and logs show application errors, retries, or slow downstream calls, the evidence points to application-level code inefficiency. If the same slow span coincides with saturated or throttled infrastructure metrics and supporting analyzer findings, the evidence points to infrastructure-level constraint.

Span structure helps narrow the application side before checking infrastructure. Custom spans represent code created inside the application, such as InvestigateTransactionDoFn:Analyze inside Dataflow (opens in a new tab) DoFn implementations, while client library spans are child spans generated by integrated clients, such as Spanner RPC queries. A dominant custom span points to owned code, while dominant client library spans point to a dependency or data-access pattern. Infrastructure analyzers then test the resource-constraint hypothesis with host CPU, host memory, HBM bandwidth, and Top-of-Rack network signals.

The CPU Utilization Analyzer detects host CPU near its allocated limit, the Host Memory Utilization Analyzer detects non-evictable memory near its limit with higher Out-of-Memory risk, the HBM Bandwidth Analyzer detects read and write throttling on TPU nodes, and the ToR NIC Link and ToR Uplink Congestion Analyzers locate switch, card, and uplink congestion for affected instances.

1.5Maintaining the Telemetry Pipeline That Enables Correlation

The telemetry pipeline that enables correlation is the path that carries OTLP trace and metric data to Google Cloud Observability. The Google-built OpenTelemetry Collector (opens in a new tab) carries that data, so its health determines whether correlation is possible. The collector emits self-observability metrics that monitor the performance and uptime of the OTLP ingestion pipeline.

Installing the OpenTelemetry Collector dashboard template from Dashboards provides at-a-glance insight into collector uptime, memory usage, and API calls to Google Cloud Observability. A gap in traces or metrics during an incident should therefore prompt a check of collector health before concluding that the application stopped emitting data. The Ops Agent with an OTLP receiver supports this comparison on Compute Engine by collecting traces and metrics through one agent, although it does not collect logs by using OTLP.

A trace waterfall visualization in Cloud Trace is a time-aligned graph of one end-to-end request and all of its measured operations. Incoming requests to Cloud Run (opens in a new tab) automatically generate a trace without additional instrumentation, and the standard W3C traceparent header is automatically populated for each request. When additional instrumentation is added, each layer the request passes through becomes a span, and the waterfall displays those spans as horizontal bars whose position and length reflect latency values.

2.1Parent-Child Span Hierarchy in a Waterfall

The trace hierarchy in a waterfall shows how one parent request contains multiple child operations. The Cloud Run request trace is the parent span that bounds the total duration, while instrumented operations inside it appear as nested child spans. Trace context propagation links spans across service boundaries, so Cloud Trace (opens in a new tab) can show a request flow across multiple services as a single trace when each service forwards the trace context.

Custom spans expose layer-specific timing that the automatic parent span cannot show alone:

• time to complete a database query
• time to receive results from an API request, such as the Cloud Translation API
• time to run complex business logic inside the service

Indentation or nesting encodes the dependency: a child span depends on its parent, and a downstream service span depends on the calling span that propagated the context to it. A gap between the end of the last child and the end of the parent represents time spent in the parent itself or in uninstrumented code, network transfer, or client-side processing.

2.2Duration, Timing, and Concurrency in a Waterfall

Duration and timing in a waterfall are read from bar position and length against a shared time axis. Each span bar starts at its start timestamp and extends for its duration, so a child that starts later appears shifted to the right of its parent start. Sequential operations appear as bars placed end to end, and their durations add directly to the parent duration because the parent cannot finish until each step completes in order.

Concurrent operations appear as overlapping bars that run in parallel during the same interval, and the parent duration is then determined by the longest overlapping branch rather than the sum of all branches. Client-side spans (opens in a new tab) are especially useful for separating these effects because they record when the client sent the RPC request and when the client received the RPC response, including latency introduced by the network and client system. Attributes, event logs, and labels attached to spans then identify which client, configuration, or downstream call each bar represents and make it easier to filter for that span in the Cloud Trace UI.

2.3Pinpointing the Service That Dominates Latency

Pinpointing the service that dominates latency means finding the blocking span on the critical path, not necessarily every long bar. A critical-path span is a child whose duration directly extends the parent end time because no other work overlaps it or because it is the longest member of a concurrent group. To find it, first identify the root Cloud Run span that defines total latency, then compare child start times and durations to separate sequential delays from parallel work, and then drill into the longest sequential child or the longest branch of overlapping children.

Interpretation must account for trace completeness because Cloud Run does not sample every request. Automatic sampling captures at most 0.1 requests per second for each instance, or about one request every 10 seconds, while a forced trace captures at most 10 requests per second for each instance. The sample rate cannot be configured for Cloud Run, so a missing slow request may simply be unsampled rather than absent from the system.

Analyzing span attributes and anomalies means reading the metadata, custom attributes, and attached logs on each span and then comparing related spans to explain errors and high latency. In Cloud Trace, a single span carries its identity and timing, its attributes describe the client and service context, and its events and logs record what happened during the operation. Anomalies become diagnosable only when those signals are correlated across a trace, across services, and against metrics.

3.1Span Metadata and Parent-Child Links

Span metadata in Cloud Trace identifies a span and places it in a trace hierarchy. Span metadata includes Span ID, Parent Span ID which is not set for a root span, Project ID that ingested the trace, and Start Time and End Time that define duration. The Trace Explorer Details flyout exposes the same identity under Metadata and Links, including span ID as a non-zero 64-bit integer, parent span ID, project ID, start and end times, and a links table that lists key-value pairs and trace IDs for links to other spans.

When a link lists Current trace, the span links to another span in the same trace, so a reader can follow parent-child and linked relationships without guessing the call path. That explicit linkage matters because trace context must be propagated correctly for Cloud Trace to assemble spans from different services into one end-to-end waterfall. Missing or broken parent links are therefore a first check when a waterfall looks fragmented.

3.2Span Attributes and Resource Context

Span attributes in Cloud Trace are key-value pairs that describe how and where the operation ran. Client-side traces separate context into groups such as client version with otel.scope.version, client environment with gcp.datastore.memory_utilization, connection properties with host, transport name such as grpc, credential type such as OAuth2, and channel flags, database properties with project ID and database ID, retry settings with initial retry delay, max attempts, max retry delay, and timeout multipliers, and OpenTelemetry configuration with otel.scope.name, service.name, and telemetry SDK language, name, and version. Resource attributes add deployment context such as cloud.account.id, cloud.provider, gcp.project_id, and App Hub identifiers like gcp.apphub.application.id, gcp.apphub.service.id, and gcp.apphub.workload.id, while operation-specific attributes add signals such as error.message, error.type, gen_ai.operation.name, gen_ai.tool.name, mcp.method.name, and host.id.

The Trace Explorer Service/workload column is set from the OpenTelemetry attribute service.name, with a fallback to the App Engine service name, so inconsistent service.name values split one logical service into multiple rows. In Observability Analytics, spans are stored in the _AllSpans view and the attributes field is type JSON, so filtering uses JSON_VALUE to extract values from that field.

• Match spans that carry any attribute with WHERE attributes IS NOT NULL.
• Match an exact attribute value with JSON_VALUE(attributes, '$.component') = 'proxy'.
• Match a substring with JSON_VALUE(attributes, '$.component') LIKE '%proxy%'.
• Resolve service identity defensively with COALESCE across resource.attributes and attributes, including fallbacks such as service.name and App Engine module keys.

Consistent service naming and defensive queries keep latency and error groupings accurate when telemetry comes from mixed SDKs and platforms.

3.3Logs, Events, and Error Signals on a Span

Logs and events in Cloud Trace attach time-stamped evidence to a specific trace_id and span_id pair. The Logs and Events tab lists only events and log entries whose trace_id and span_id match the selected span, and a circle on the waterfall latency bar marks spans that have an associated log or event. The tab also offers View logs to open the same trace, span, and time range in Logs Explorer, while separate Stacktraces and Inputs/Outputs tabs expose stacktraces and generative AI events that follow OpenTelemetry GenAI conventions.

Status and error attributes complete the signal because status.code, error.type, and error.message distinguish failed operations from slow but successful ones. Database client traces (opens in a new tab) show how specific those events can be, with Lookup, Commit, and RunQuery operations each emitting a completion event with overlapping but distinct fields.

A span with high duration but normal doc_count points in a different direction than a span with large doc_count, missing keys, or unfinished query results, so the event fields narrow whether the anomaly looks like contention, an oversized read, or an incomplete result set.

3.4Aggregating Spans to Surface Latency and Error Anomalies

Aggregation in Observability Analytics and Trace Explorer turns individual slow spans into patterns of latency outliers and error rates. The Spans table lists individual spans and can be sorted by duration to find high-latency outliers, while the Grouped table groups spans by span name and service or workload name to show average latency and statistical behavior. The heatmap uses darker cells for time intervals with many spans, the Span duration percentile chart tracks the 50th, 90th, 95th, and 99th percentiles over time, and the Span rate chart shows response status as a function of time.

Span data is stored for 30 days, and selecting a heatmap cell filters the tables to that interval, so a spike can be isolated before opening the trace waterfall. SQL over _AllSpans (opens in a new tab) supports the same percentile and grouping logic at scale, including grouping by hour and status code, counting by status or operation name, and averaging duration to rank the slowest operations. Per-service statistics combine COUNT(*), AVG(duration_nano), MIN(duration_nano), MAX(duration_nano), and APPROX_QUANTILES(duration_nano, 100)[OFFSET(50)], [OFFSET(95)], and [OFFSET(99)] to compute p50, p95, and p99 latency alongside span count and unique trace IDs. If the query includes start_time in the WHERE clause, that clause overrides the time-range selector, so a query that filters the last hour ignores a selector set to the last 7 days.

3.5Correlating Anomalies Across Spans to Find Root Cause

Correlation across spans distinguishes a slow dependency from a broken trace view. A complete diagnosis first checks whether the trace itself is incomplete, then checks whether related spans share the same latency, error, or resource signature. An incomplete trace appears with a Missing span ID message when a sampled span references another span that was not sampled, was sampled but not yet received, or was received but not stored.

Persistent gaps point to sampling and propagation rather than application latency, so components under control should respect and propagate the W3C sampled flag hint, Service Mesh trace context propagation should be verified, and Google Cloud rate limits on trace writes should be considered. Other incomplete views come from searching the wrong projects, lacking the Cloud Trace User role roles/cloudtrace.user on projects that store span data, or querying buckets in multiple regions, which fails with a cross-regional query error. Troubleshooting incomplete traces (opens in a new tab) therefore comes before concluding that a service is slow.

Once completeness is confirmed, correlate duration, status, attributes, and metrics across parent and child spans in order:

1. Confirm the slow edge: Open the trace waterfall or directed acyclic graph view, where bar length shows duration and color shows status, and identify which child span contributes most of the parent duration.
2. Read the child context: Inspect that span's attributes for host, database ID, retry settings, service name, and error fields, plus its Lookup, Commit, or RunQuery event counts and consistency setting.
3. Check sibling and downstream spans: Compare spans with the same name and service in the Grouped table and percentile charts to decide whether the delay is isolated or systemic.
4. Validate against resource metrics: For Cloud Spanner (opens in a new tab) latency, use Cloud Monitoring (opens in a new tab) to chart spanner.googleapis.com/api/request_latencies against CPU utilization on a log-scale Y-axis, then check query statistics and query execution plans during the same interval.

That last comparison separates two documented Cloud Spanner root causes. If CPU utilization exceeds the recommended maximum and latency is high without many CPU-intensive queries, the instance needs more compute capacity to handle the workload. If the same high-CPU and high-latency window contains queries with many joins, missing query parameters, missing secondary indexes, or other CPU-intensive operations, the fix is to rewrite queries under SQL best practices and revise schema design for more efficient access.

Key Facts

• A trace waterfall displays one request as nested parent and child spans, where sequential bars add directly to parent duration and concurrent overlapping bars contribute only the longest branch.
• Cloud Run automatic sampling captures at most 0.1 requests per second per instance and forced traces capture at most 10 requests per second, and the sample rate cannot be configured.
• Correlation requires traces for where time was spent, Cloud Monitoring metrics for resource saturation such as CPU throttling or memory pressure, and Cloud Logging entries matched by trace ID and span ID.
• Span data is stored for 30 days, and Observability Analytics stores spans in the _AllSpans view where the attributes field is queried with JSON_VALUE.
• For Cloud Spanner, high latency with CPU above the recommended maximum and no costly queries indicates a need for more compute, while the same signals with joins, missing parameters, or missing indexes indicate a need to rewrite queries and revise schema design.