Professional Cloud DevOps Engineer

Analyze Span Attributes and Anomalies

When you examine a distributed trace in Cloud Trace, you look at individual time measurements called spans. Each span holds metadata such as the service name, span name, status, duration, and span kind. The span kind tells you how spans relate to each other (for example, a client span calls a server span). By evaluating these span attributes, you can spot anomalies like high latency or error statuses. Custom attributes—such as user IDs or request parameters—can be added to spans to give extra context for diagnosing issues.

Cloud Trace provides visualization tools to help you find unusual patterns. The heatmap uses color intensity to show where most spans fall, with darker colors meaning more spans. You can also view latency percentiles (50th, 90th, 95th, and 99th) to understand the spread of response times. The Span rate chart shows how many spans are being sent over time, which helps you detect sudden traffic changes that might signal a problem.

To filter and correlate trace data, use the Span filters pane. It offers common filters like OpenTelemetry service, span name, span status, duration, and span kind. You can also create custom filters in the Filter bar, for example filtering by HTTP status code (/http/status_code: 200) or host. When troubleshooting, search for a specific trace ID to explore an individual trace in detail. Correlate anomalies across related spans to diagnose root causes such as network delays, database contention, or third-party API failures.

Start your performance analysis by sorting the Spans table by duration to find the slowest spans. The Grouped table shows average latency grouped by span name or service. Look for patterns: long wait times in database spans suggest database contention, spans representing external API calls with high latency indicate network delays, and error statuses in those spans point to third-party API failures. By correlating these patterns across multiple traces, you can systematically identify and resolve performance bottlenecks.

Interpret Trace Waterfall Visualizations

A trace waterfall is a hierarchical graph that shows how a request moves through multiple services. Each span represents the time a request spends in a specific layer, such as a database query or an external API call. The waterfall displays parent-child relationships: a parent span might represent the overall request, while child spans represent sub-operations. By decoding this hierarchy, you can see which operations run sequentially and which run concurrently, and pinpoint the single span that contributes most to total latency.

For a trace waterfall to be complete, services must pass trace information to downstream operations. This is called context propagation, and it relies on standardized HTTP headers like the W3C traceparent header. Context propagation ensures that multiple service calls are grouped into one cohesive trace, rather than appearing as isolated segments. It is important to distinguish context propagation from sampling: sampling decides whether a component records data about a span, while context propagation only transfers identifiers like the span ID to child components.

The completeness of a trace waterfall also depends on the configured sampling strategy. You must evaluate different options to optimize telemetry collection. Head-based sampling makes the sampling decision when the request is first received. Tail-based sampling delays the decision until the entire trace is complete, using an intermediary OpenTelemetry Collector. Ensuring complete traces is critical because missing spans can hide the real cause of latency.

When analyzing a waterfall graph, assess the timing of operations. If overall latency is high but database metrics are low, the bottleneck likely resides in the application code or in resource limits of the parent service. By identifying the specific span that contributes most to total latency, you can isolate the root cause and focus your debugging efforts.

Correlate Traces with System Metrics and Logs

Correlating traces with metrics and logs is essential for root cause analysis in distributed applications on GCP. By linking data from Cloud Trace, Cloud Monitoring, and Cloud Logging, you can move from seeing a symptom (like high latency) to understanding its cause. This integrated view helps you distinguish between application-level code inefficiencies (for example, a slow database query) and infrastructure-level resource constraints (such as CPU throttling or memory pressure).

The process begins in the Trace Explorer, where you analyze spans and trace waterfalls. Within a selected trace, you can examine specific spans to see request details like HTTP method and status code. The Logs & Events tab in the Trace Explorer displays logs directly associated with that trace, giving you contextual application events. Conversely, in the Logs Explorer, you can start from a log entry and use the Traces icon to jump to the related trace details. This creates a bidirectional investigative flow: you can go from trace to logs or from logs to trace.

To contextualize performance issues on a dashboard, create a customized dashboard in Cloud Monitoring that combines relevant metrics (like request latency or error rates) with log panels and trace sample data. For example, you can correlate a spike in backend latency metrics with specific error logs and the corresponding slow trace. This reveals whether the bottleneck is in your code, a downstream service, or the underlying infrastructure.

Advanced troubleshooting requires correlating system metrics—such as high CPU usage or memory pressure from Cloud Monitoring—with trace data. If a trace shows increased latency in a specific microservice, check the associated host's CPU metrics to see if the issue is due to resource saturation. Similarly, correlate trace data with infrastructure logs from GKE or Compute Engine to uncover issues like pod evictions or network errors that impact application performance. This completes the diagnostic picture, allowing you to pinpoint the exact cause of a performance problem.