Analyzing Distributed Traces with OpenTelemetry and Cloud Trace

Utilizing tracing frameworks means adding instrumentation code to applications so each request generates spans and traces that flow to a Google Cloud project for storage in Cloud Trace (opens in a new tab). OpenTelemetry (opens in a new tab) is the recommended vendor-neutral framework for this work, and telemetry flows from the application process to the project either through an intermediate collector or directly through the Telemetry (OTLP) API. Before any flow starts, the Cloud Logging, Cloud Monitoring, and Cloud Trace APIs must be enabled in the project, which requires the serviceusage.services.enable permission through the Owner role or the Service Usage Admin role.

1.1When Manual Instrumentation Is Required

Manual instrumentation is required when automatic trace capture does not cover a request path that must be validated or troubleshot. Some Google Cloud services capture trace data automatically, so those workloads may not need added code, while other paths leave gaps that hide service interactions. In-process instrumentation runs inside the same application process, observes application logic, and generates logs, metrics, and traces that complement system metrics such as CPU, memory, and disk usage. This application telemetry adds error messages, stack traces, source-code locations, and the service-to-service calls that make up one request.

1.2Why OpenTelemetry Is the Standard Choice

OpenTelemetry is the vendor-neutral standard for generating distributed traces because it is not tied to one backend and defines its own data model. A vendor-neutral framework lets the destination change without rewriting application code, and it provides a consistent approach across supported languages. It also provides interoperable libraries for traces and metrics, so teams can use trace libraries, metric libraries, or both, and can add custom application-specific measurements. An instrumentation approach (opens in a new tab) that uses OpenTelemetry therefore keeps telemetry comparable across all services.

1.3Generating Spans and Custom Traces

Custom traces (opens in a new tab) in OpenTelemetry are built from spans, where each span represents one unit of work such as a function call or set of subrequests. A developer first acquires a Tracer with a name that identifies the instrumentation scope, then starts a new span from the current request context with an operation name such as subrequests. The span stays active while the work runs and must call End() before the function exits, because only completed spans are exported. OpenTelemetry libraries can also generate telemetry for popular frameworks automatically, while custom instrumentation records behavior that automatic libraries cannot infer.

1.4Choosing a Collector-Based or Direct Export Path

The export path is the route that moves telemetry from the OpenTelemetry SDK to the Google Cloud project using the OpenTelemetry Protocol (OTLP) wire format. The Telemetry (OTLP) API implements OTLP as the ingestion endpoint in the project, and the OpenTelemetry Collector configuration file controls how a collector receives, processes, and exports telemetry when a collector is used.

Use collector-based export when the environment can run a separate collector process, and use direct export when running a separate collector is not practical. Existing instrumented applications can point at a collector by setting the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to the collector OTLP endpoint, such as the in-cluster address on port 4317.

1.5Platform Collectors and Managed Collection Patterns

Platform collectors are Google Cloud agents and managed endpoints that receive OTLP telemetry and forward it to the project, and the choice depends on the compute environment. The Google-built OpenTelemetry Collector receives OTLP from SDK exporters and exports traces to Google Cloud Observability, and on Google Kubernetes Engine (GKE) it sends traces through the Telemetry API and metrics to Google Cloud Managed Service for Prometheus. The Ops Agent is the primary agent for Compute Engine that receives OTLP metrics and traces, then sends metrics through the Cloud Monitoring API and traces through the Cloud Trace API. Managed OpenTelemetry for GKE (opens in a new tab) provides a managed in-cluster OTLP endpoint without operating a collector and is the only Google-managed solution for trace collection on GKE.

1.6Controlling Sampling, Access, and Ingestion Cost

Sampling strategy determines which traces are created, propagated, and exported so high-traffic services keep insight without exporting every request. The SDK makes the initial decision, the sampled flag in the W3C trace context propagates it across services, and probability sampling, tail sampling in the collector, span-name rates, and maximum-rate enforcement provide complementary controls. Collection of spans and use of the OpenTelemetry framework are free, while ingestion into Cloud Trace is billable, so teams start with a small ratio and use collector processors to drop, batch, and filter before charges apply. Access is controlled by Identity and Access Management roles where roles/cloudtrace.user permits reading traces, roles/cloudtrace.agent permits writing traces, and roles/cloudtrace.admin permits both.

Analyzing trace waterfalls and spans means turning distributed timing records in Cloud Trace (opens in a new tab) into a diagnosis of why a request was slow or failed. A trace represents one end-to-end request and is composed of spans where each span measures time in one layer such as a database query, API call, or business logic. Analysis reconstructs span order across services, finds which span determined total latency, and connects timing to logs and metrics to locate the root cause.

2.1What Metrics, Logs, and Traces Each Contribute

Metrics, logs, and traces answer different questions about the same workload and must be used together for correlation. Metrics are numerical measurements collected at intervals such as request rate, error rate, response time, CPU, network I/O, disk I/O, and business measures such as orders placed. Logs are time-stamped records of discrete events such as failures, errors, and state changes, and traces show the components used in a request journey, where bottlenecks exist, and how long each span took. Default service metrics reveal component dependencies, custom metrics extend monitoring to application behavior, and Active Assist analyzes utilization trends to recommend resource adjustments.

2.2Parent-Child Hierarchy, Timing, and Concurrency in Waterfalls

The waterfall hierarchy shows how one parent request contains multiple child operations ordered by trace context propagation. The Cloud Run request trace is the parent span that bounds total duration, while instrumented operations such as a database query, a Cloud Translation API call, or complex business logic appear as nested child spans. Each bar starts at its start timestamp and extends for its duration, so sequential bars placed end to end add directly to parent duration because the parent cannot finish until each step completes. Concurrent bars overlap in parallel and the parent duration is then set by the longest overlapping branch, while a gap after the last child represents time in the parent itself, uninstrumented code, network transfer, or client processing.

2.3Pinpointing the Service That Dominates Latency

Pinpointing latency means finding the blocking span on the critical path, which is the longest dependent chain that determines total request duration. To find it, first identify the root Cloud Run span that defines total latency, then compare child start times and durations to separate sequential delays from parallel work, and then drill into the longest sequential child or longest overlapping branch. Optimizing a span off that chain does not reduce end-to-end latency, while shortening a span on the critical path directly reduces it. Interpretation must account for sampling because Cloud Run standard sampling captures at most 0.1 requests per second per instance and forced tracing raises the limit to about 10 requests per second per instance, and the rate cannot be configured.

2.4Moving Between Trace Waterfalls and Log Entries

The Trace explorer to Logs Explorer pivot connects a slow span to events from the same request through shared trace context. Trace data flows from instrumented applications through the Telemetry (OTLP) API into the project, where Trace explorer shows a table of traces and a Gantt chart of spans in the Trace details panel. To move from trace to logs, follow this order:

1. Select the trace: In Trace explorer, select the table row for the request of interest, such as span name /multi.
2. Inspect the span: In the Gantt chart, select the span to open request details including method, status code, bytes, and caller user agent.
3. Open associated logs: Select the Logs and Events tab to see entries with the same trace context, expand an entry, or select View Logs to continue in Logs Explorer.

The reverse pivot starts from a log entry such as handle /multi request, selects Traces on an entry that carries trace context, and selects View trace details to reopen the same waterfall.

2.5Span Attributes, Events, and Aggregation for Anomalies

Span attributes are key-value pairs that describe how and where an operation ran, including client version, environment, connection, database, retry settings, and resource context such as service.name, cloud.provider, and gcp.project_id. The Service/workload column is set from service.name with fallback to the App Engine service name, so inconsistent naming splits one service into multiple rows. In Observability Analytics, spans are stored in the _AllSpans view with attributes as JSON queried with JSON_VALUE, and span data is stored for 30 days. Aggregation surfaces patterns through the Spans table sorted by duration, the Grouped table by span and service name, a heatmap for dense intervals, percentile charts for p50, p90, p95, and p99, and SQL with COUNT(*), AVG(duration_nano), and APPROX_QUANTILES to rank slowest operations.

2.6Distinguishing Code Inefficiency from Resource Constraint

Distinguishing code inefficiency from resource constraint requires comparing span structure against infrastructure saturation in the same interval. Custom spans such as InvestigateTransactionDoFn:Analyze in Dataflow (opens in a new tab) DoFn implementations point to owned code, while client library spans such as Spanner RPC queries point to a dependency or data-access pattern. If one span dominates with flat CPU and memory and logs show errors or retries, the evidence points to application code, while saturation or throttling during the same slow span points to infrastructure.

The CPU, host memory, HBM Bandwidth, and Top-of-Rack congestion analyzers test the resource hypothesis, and the Google-built collector (opens in a new tab) health must also be checked because its self-observability metrics determine whether missing traces reflect pipeline failure. For Cloud Spanner latency (opens in a new tab), high latency with CPU above the recommended maximum and no costly queries indicates a need for more compute, while the same signals with joins, missing parameters, or missing indexes indicate a need to rewrite queries and revise schema design.

Correlating trace IDs with structured logs means carrying the identity of one distributed request across services and embedding that identity in logs so errors and latency can be reviewed together. Cloud Trace (opens in a new tab) captures the end-to-end path and latency as spans, while Cloud Logging stores timestamped log entries with event detail. Correlation links those two views through a shared trace identifier, so filtering by trace ID isolates every event for one transaction and opening the same ID in Cloud Trace shows latency across each span.

3.1How Trace Context and Structured Logs Enable Correlation

Trace context and structured logs enable correlation because traces show request flow while logs show event detail. A trace represents the path of a single end-to-end operation across distributed components, and each trace is composed of spans that record one function with latency data. A log is a collection of timestamped entries where each entry describes a specific event, but logs alone do not show how a change in one component relates to others. Added log fields such as severity, service name, trace ID, span ID, and customer ID make troubleshooting faster and allow log entries in Logs Explorer to link directly to the matching trace.

3.2Instrumentation Options for Propagating Trace Context

Instrumentation for propagating trace context centers on OpenTelemetry as the recommended vendor-neutral way to create spans, propagate context on outbound calls, and export telemetry to the project. When trace data is not automatically captured, the application must be instrumented to create spans, forward context, and export to the destination project where traces become viewable alongside logs and metrics. Data flows from the instrumented application to the export destination through peer paths that keep the same project destination, including OpenTelemetry with OTLP to a collector that forwards through the Telemetry API, OpenTelemetry with OTLP directly through the Telemetry API, Ops Agent with OTLP receiver on Compute Engine, direct Telemetry or Cloud Trace API invocation, Spring Boot forwarding, and Cloud Trace client libraries.

3.3Configuring Compute Platforms to Collect Correlated Telemetry

Configuring compute platforms means combining what each platform emits by default with explicit collection for custom spans and trace-enriched logs. Compute Engine, Google Kubernetes Engine (GKE), and Cloud Run generate system metrics and logs by default, while Cloud Run functions and App Engine automatically generate metrics, logs, and traces. Additional configuration is required when custom spans, application logs with trace fields, or user-defined metrics must flow to the same project.

The Ops Agent automatically collects host CPU, GPU, memory, process metrics, syslog or Windows Event Log, third-party integrations, Prometheus metrics, and OTLP traces and metrics, while application logs use logging receivers rather than OTLP.

3.4Structuring Logs to Carry Trace IDs and Enforcing Standards

Structuring logs to carry trace IDs turns isolated entries into queryable steps of a distributed trace by preserving the trace identifier, span identifier, and metadata such as service name and severity. Application logs reach Cloud Logging by writing to files such as syslog and forwarding with the Ops Agent, by configuring packages such as fluentd, Logback, Winston, or standard language facilities, or by calling the Logging API (opens in a new tab) directly. Entries remain correlatable only with valid timestamps, because entries older than the retention period or more than 24 hours in the future are discarded. Organizational standards extend this by mandating structured formats, requiring timestamps, success status, service-account identity with serviceAccountDelegationInfo, and CI/CD identifiers, enabling Data Access audit logs (opens in a new tab) for IAM and token services, and routing copies via sinks to BigQuery, Cloud Storage, and Pub/Sub for validation.

3.5Querying Correlated Data with Observability Analytics and BigQuery

Querying correlated data means using the shared trace identifier across tools optimized for single-request inspection versus aggregate trends. Logs Explorer reads individual log entries, Trace Explorer inspects latency for one request, Observability Analytics runs SQL over log views and the _AllSpans trace view to aggregate trends, and BigQuery Studio through a linked dataset joins telemetry with business data. Libraries such as Sqlcommenter extend correlation to databases by augmenting SQL with tags and propagating OpenTelemetry context to the database. A linked dataset is required only to join with other datasets, query from BigQuery Studio, use reserved slots, or alert on log query results, and log buckets must have analytics enabled to support SQL analysis.

Employing Gemini Cloud Assist (opens in a new tab) for trace analysis means using conversational chat and deeper automated Investigations to correlate logs, metrics, configurations, and trace signals into ranked explanations. Chat provides immediate explanations of an error or latency symptom, while Investigations performs automated comparison of baseline behavior against an anomalous period. Data flows from the selected resource and its observability stores to the Cloud Assist panel or Investigation engine, and results flow back as structured output that must still be validated before changing production.

4.1Configuring and Integrating Gemini Cloud Assist with Tracing Systems

Configuration starts with project setup, least-privilege Identity and Access Management (IAM) roles, and enabling backing APIs so Gemini can only analyze telemetry the user is authorized to view. An administrator ensures Gemini Cloud Assist is set up for the account and project and waits for propagation, then the user opens chat from any console page and prompts from the resource page so the response includes current page and project context. The documented role pattern pairs Monitoring Viewer, a service-data reader such as Bigtable Reader, Database Insights Viewer, and Investigation Owner or Creator, with additional APIs such as Service Health, Recommender, and App Optimize for their domains. As of April 10, 2026, creating, running, and editing investigations additionally requires a Premium Support contract, while prior results remain visible.

4.2Formulating Context-Rich Prompts for Trace Patterns

Crafting prompts for trace patterns means giving Gemini the exact resource, time window, and trace evidence needed to separate high latency, error rates, and dependency failures. Generic questions return general guidance, while prompts with service name, cluster or project, status code or exception text, and latency pattern return targeted next steps. A context-rich prompt includes resource identity such as the api-gateway service in prod GKE, trace evidence such as HTTP 429 or panic: runtime error, performance scope such as elevated 5XX rate or CrashLoopBackoff, and dependency context such as recent deployments, ingress settings, instance limits, or baseline comparison window. Navigating to the resource page before prompting automatically scopes the response, but responses are time-bounded such as to the last hour in database-assisted flows.

4.3How Investigations Produce Ranked Recommendations

Investigations produce recommendations through automated correlation of baseline aggregated data against the selected anomalous period for logs, configurations, and metrics. When launched from Logs Explorer, Bigtable System insights, Cloud Run (opens in a new tab), or GKE workflows (opens in a new tab), data flows into the investigation engine and results return to the Investigation Overview pane. For Bigtable, the engine compares baseline against client latencies, server latencies, CPU utilization, disk utilization, and request count to detect significant change.

In some situations analysis completes without a recommendation, which signals that data does not support a confident conclusion. For source-deployed services, Cloud Run investigations can also send code fix suggestions to the Source tab.

4.4Validating Findings and Rolling Out Optimizations Safely

Validation is required because Gemini can misinterpret errors, miss hidden patterns, or suggest actions that do not fit the environment. Validation means treating Issue, Findings, and Hypotheses as starting points and confirming that time window, resource, cluster, and symptoms match monitoring, with credible findings showing movement in more than one dimension such as rising client and server latencies with rising CPU and request count. A hypothesis about resource constraints should align with container or database telemetry, while network congestion should align with span durations and request logs. A disciplined rollout then applies one change at a time:

1. Isolate one hypothesis: Select a single action and define expected change in latency, error rate, CPU, or throughput.
2. Apply to a subset first: Roll to a canary group, single cluster, or A/B cohort while remaining traffic serves as control.
3. Measure SLIs against baseline: Compare latencies, utilization, and success rate for test versus baseline and control over the same period.
4. Promote or roll back: Promote broadly only after service-level indicators improve without objective regression, otherwise roll back.
5. Escalate with context if needed: Transfer results to a support case to provide environment state and log summaries.

Key Facts

• OpenTelemetry spans are created by a Tracer with an instrumentation scope name and only completed spans that call End() are exported to Cloud Trace.
• Collector-based export keeps application code vendor-neutral by holding Google Cloud configuration in the collector, while direct export sends OTLP directly through the Telemetry API without a collector.
• Cloud Run standard sampling captures at most 0.1 requests per second per instance and forced tracing raises the limit to about 10 requests per second per instance.
• Correlation requires a shared trace ID across Cloud Trace spans and Cloud Logging entries, with Observability Analytics querying the _AllSpans view and a linked BigQuery dataset required only for joins, external queries, reserved slots, or alerting.
• Gemini Cloud Assist Investigations returns Issue with start and stop time, Findings with observations, and Hypotheses with remediation actions, and creating investigations requires a Premium Support contract as of April 10, 2026.