Distributed Tracing with OpenTelemetry on Google Cloud

Instrumenting applications for telemetry export means adding code that generates spans and traces inside the application and then sending that data to a Google Cloud project for storage and analysis. The recommended path uses Cloud Trace (opens in a new tab) as the storage and analysis backend and OpenTelemetry (opens in a new tab) as the vendor-neutral framework that creates the telemetry. Telemetry then flows from the application process to the project either through an intermediate collector or directly through the Telemetry (OTLP) API.

1.1When Manual Instrumentation Is Needed

Manual instrumentation is needed when automatic trace capture does not cover a request path that must be validated or troubleshot. Some Google Cloud services and configurations capture trace data automatically, so those workloads may not need added code. When automatic tracing leaves gaps, developers add an instrumentation approach (opens in a new tab) to collect language runtime behavior, framework activity, and application logic. This application-level telemetry complements system metrics such as CPU, memory, and disk usage by showing error messages, stack traces, source-code locations, and how services interact to process one request.

1.2How In-Process Instrumentation Works

In-process instrumentation is code that runs inside the same application process, observes application logic, and generates logs, metrics, and traces. Data flows from the application logic to the instrumentation framework inside the process, and then the framework exports the telemetry from the process to a configured storage system such as a Google Cloud project. Google Cloud Observability then collects, analyzes, and correlates that telemetry and provides defaults such as dashboards and alerting policies. Before any flow can start, the Cloud Logging, Cloud Monitoring, and Cloud Trace APIs must be enabled in the project, which requires the serviceusage.services.enable permission through the Owner role or the Service Usage Admin role.

1.3Why OpenTelemetry Is Recommended

OpenTelemetry is the recommended vendor-neutral framework for generating distributed traces, even when telemetry is sent only to Google Cloud. A vendor-neutral framework is not tied to one vendor and defines its own data model, so the destination can usually change without rewriting application code. OpenTelemetry provides a standardized approach across supported languages, so telemetry from all services can be collected and compared consistently. It also provides interoperable libraries for different signals, so teams can use trace libraries, metric libraries, or both, and can add custom application-specific measurements on top of framework-generated telemetry.

1.4Generating Spans and Custom Traces

Custom traces (opens in a new tab) in OpenTelemetry are built from spans, where each span represents one unit of work such as a function call or a set of subrequests. A developer first acquires a Tracer, which is the creator of spans, with a name that identifies the instrumentation scope. The tracer then starts a new span from the current request context, for example tracer.Start with a span name such as subrequests, keeps the span active while the work runs, and calls End() right before the function exits. Only completed spans are exported, so an unended span is never sent to the backend.

For companion signals, Google Cloud recommends JSON-structured logs for Cloud Logging using Go slog, Python logging, JavaScript Pino, and Java SLF4J with Log4j2. For metrics, Google Cloud recommends OpenTelemetry or Prometheus client libraries that expose metrics at an HTTP endpoint for scraping. OpenTelemetry instrumentation libraries can generate telemetry for popular frameworks automatically, while custom instrumentation records additional application-specific behavior that automatic libraries cannot infer.

1.5Choosing an Export Path for Telemetry

The export path is the route that moves telemetry from the OpenTelemetry SDK in the application to the Google Cloud project. The OpenTelemetry Protocol (OTLP) is the recommended wire format, and the Telemetry (OTLP) API implements OTLP as the ingestion endpoint in the project. The OpenTelemetry Collector (opens in a new tab) configuration file controls how a collector receives, processes, and exports telemetry when a collector is used.

Use collector-based export when the compute environment can run a separate collector process, and use direct export when running a separate collector is not practical. Other supported paths include invoking the Telemetry API or Cloud Trace API directly, forwarding Spring Boot trace data to Cloud Trace, and using Cloud Trace client libraries or the Cloud Trace exporter for OpenTelemetry.

1.6Platform Collectors and Managed Collection Patterns

Platform collectors are the Google Cloud agents and managed endpoints that receive OTLP telemetry and forward it to the project, and the choice depends on the compute environment. All patterns preserve the same order: first instrument with the OpenTelemetry SDK and OTLP exporter, then receive telemetry in a collector or agent, then export to the project. Existing instrumented applications can be pointed at a collector by setting the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to the collector OTLP endpoint, such as the in-cluster collector address on port 4317.

• Google-built OpenTelemetry Collector: Google's open-source build of the OpenTelemetry Collector that receives OTLP telemetry from SDK exporters and exports it to Google Cloud Observability. On Google Kubernetes Engine (GKE), the collector receives trace data from the SDK in-process OTLP exporter, processes it, and sends it to the project through the Telemetry (OTLP) API. For metrics on GKE, the collector can export to Google Cloud Managed Service for Prometheus.
• Ops Agent: The primary telemetry agent for Compute Engine instances that receives OTLP metrics and traces from SDK exporters, transforms the data, and sends metric data through the Cloud Monitoring API and trace data through the Cloud Trace API. The Ops Agent can also collect logs, but not by using OTLP, and it can collect Prometheus-format metrics through a Prometheus receiver.
• Managed OpenTelemetry for GKE: A managed in-cluster OTLP endpoint that receives OTLP traces, metrics, and logs without requiring teams to manage, authenticate, upgrade, or monitor their own collector. It is the only Google-managed solution for collecting trace data on GKE and requires workloads to already be instrumented with the OpenTelemetry SDK using standard OpenTelemetry environment variables.

Managed OpenTelemetry for GKE also supports automatic configuration through an Instrumentation custom resource that enables generation and ingestion of correlated traces, metrics, and logs. That resource supports the Agent Development Kit and multimodal prompt and response data, and it is distinct from the Instrumentation resource of the OpenTelemetry operator. If collector-level filtering and controls are required, use the Google-built collector instead of the managed service. On GKE, applications should output JSON-structured logs to stdout and stderr for automatic collection, while on Compute Engine applications should output JSON-structured logs to a file collected by the Ops Agent.

Analyzing trace data to diagnose performance and errors is the practice of turning distributed timing records into a diagnosis of why a request was slow or failed. A trace in Cloud Trace (opens in a new tab) represents one end-to-end request, and it is composed of spans where each span measures time spent in one layer or operation such as a database query, an API call, or business logic. Analysis reconstructs the order of spans across services, finds which span determined total latency, and connects that timing evidence to logs and metrics to locate the root cause.

2.1Reconstructing End-to-End Request Flows Across Services

Request-flow reconstruction is the joining of spans from different microservices into a single trace by propagating trace context from caller to callee. OpenTelemetry is the recommended vendor-neutral framework for this work, and propagation on Google Cloud commonly uses the W3C traceparent header to carry the trace identifier across service boundaries. Incoming requests to Cloud Run (opens in a new tab) automatically generate a root trace and automatically populate traceparent without code changes, so an initial flow is visible immediately.

Automatic spans alone show only the front-end boundary, so full reconstruction depends on instrumentation inside each service. OpenTelemetry libraries automatically create spans at remote procedure call (RPC) boundaries, while custom spans must be added explicitly to expose internal work such as a call to the Cloud Translation API. When every service forwards the incoming context and emits its own spans to the same trace, Cloud Trace renders the result as a waterfall graph where parent spans contain child spans in execution order. If child spans are missing or appear as separate traces, trace context was not propagated between those two services.

2.2Identifying Latency Outliers and Critical Paths

Latency analysis in Cloud Trace compares individual request timing against aggregate behavior to separate systemic slowness from outliers. The waterfall view shows how long each span took and how spans overlapped, which reveals the critical path as the longest dependent chain of spans that determines total request duration. Optimizing a span off that chain does not reduce end-to-end latency, while shortening a span on the critical path directly reduces it.

Outliers require attention to sampling because Cloud Run does not trace every request. Standard sampling on Cloud Run captures at most 0.1 requests per second per instance, or about one request every 10 seconds, and the rate cannot be configured. A specific request can be forced to trace, which raises the limit to about 10 requests per second per instance, but forced traces still do not capture every request. Because rare slow requests may never be sampled, add span labels to custom spans to make slow dependencies filterable, and use forced tracing when reproducing a specific failure or tail-latency case.

2.3Correlating Traces with Logs, Metrics, and Span Details

Correlation is the connection of a slow or failed span to the logs, metrics, and attributes that explain why it behaved that way. Client-side traces collected from RPC execution are especially useful because they capture what the application experienced, not only what the server recorded:

• Spans carry send and receive timestamps that include network and client-system latency in addition to server time.
• Attributes carry key-value details about the client and its configuration.
• Event logs record key events associated with a span.
• Stack traces capture client-side crash context when an exception occurs.

A gap between client-observed latency and server-observed latency therefore points to network transit, retries, or client overhead rather than server processing. Attributes and labels then narrow the comparison by version, instance, database, or operation, while linked log events and exception stack traces identify the failing call and error. For broader validation, compare the trace finding against Cloud Monitoring signals such as CPU utilization, memory usage, network traffic, disk I/O, and application response times, and against service level objectives (SLOs) such as response latency at a given percentile.

2.4Controlling Sampling, Export Path, and Access for Reliable Analysis

Reliable analysis depends on getting enough complete traces to the backend without overwhelming cost or the application. Traces for Firestore (opens in a new tab) and Firestore in Datastore mode (opens in a new tab) client libraries are instrumented with OpenTelemetry APIs, but coverage has limits: trace spans are available only for the Java and Node.js client libraries in Firestore Native mode and only for the Java client library in Datastore mode, and no spans are produced for real-time listeners. Collection itself and use of the OpenTelemetry framework are free, while ingestion of spans into an observability backend such as Cloud Trace is billable, so teams typically start with a small sampling ratio based on traffic.

Data flows from the application to the backend through one of two export paths, and the service account that writes traces must hold permission to write traces while the Trace API is enabled on the project.

Access to stored traces is controlled by Identity and Access Management roles on the writing and reading projects. The roles/cloudtrace.user role permits reading traces, the roles/cloudtrace.agent role permits writing traces, and the roles/cloudtrace.admin role permits both reading and writing.

Optimizing a tracing strategy for cost and observability means controlling how much trace data flows from instrumented applications to a Google Cloud project while preserving the detail needed to meet service-level objectives (SLOs). In Google Cloud, OpenTelemetry (opens in a new tab) generates spans and traces in the application process, an OpenTelemetry Collector or direct exporter sends that telemetry through the Telemetry (OTLP) API, and Cloud Trace (opens in a new tab) stores and analyzes the ingested spans. Because billing is by ingestion volume and Cloud Trace quotas and limits apply, every sampling, attribute, and collector decision directly changes cost, query value, and reliability signal.

3.1Sampling Strategies to Balance Volume with Insight

A sampling strategy (opens in a new tab) determines which traces are created, propagated, and exported, so high-traffic services keep representative insight without exporting every request. The application SDK makes the initial sampling decision, the sampled flag in the W3C trace context propagates that decision across service boundaries, and the collector or backend receives only the sampled spans. When trace data for validating performance or troubleshooting is not automatically captured, instrumented sampling becomes the control for trading completeness against ingestion cost and quota consumption.

OpenTelemetry-based workflows support several complementary sampling controls:

• Probability sampling: keeps a configured percentage of traces at the SDK, reducing volume in a statistically predictable way.
• Tail sampling processor: makes a keep-or-drop decision in the OpenTelemetry Collector after a full trace is seen, so rare errors and slow requests can be retained while routine traffic is dropped.
• sampled flag propagation: carries the head-based decision from the originating service to downstream services, preserving end-to-end continuity of a single request flow.
• Span-name and service-specific rates: adjusts sampling by operation, such as platform settings like the Cloud Endpoints trace sampling rate or a Jaeger remote sampling strategy keyed by span name.
• Maximum sampling rate enforcement: caps the rate a Google Cloud service will generate, with additional support requested through Google Issue Tracker.

Choose probability or rate limits when steady-state cost control matters most, and choose tail sampling in the collector when diagnostic value depends on outcome, such as keeping failed traces and discarding successful ones.

3.2Defining Meaningful Custom Spans, Attributes, and Events

Custom instrumentation (opens in a new tab) adds application-specific detail that automatic tracing cannot infer, such as domain operations, framework behavior, and language-runtime context. In OpenTelemetry, spans are the building blocks of traces, a Tracer acquired for an instrumentation scope creates spans, and attributes and events attached to those spans explain what happened inside the application during that operation. Well-chosen names and attributes let Cloud Trace group by service and operation, expose latency for individual requests and aggregate latency across services, and correlate traces with logs and metrics that share the same context.

Create a custom span through an ordered workflow where each step depends on the previous one:

1. Acquire a Tracer: obtain an OpenTelemetry Tracer with an instrumentation scope name that identifies the library or component generating the traces.
2. Start a span: call tracer.Start with the parent context and an operation name, such as subrequests, so downstream calls made with the returned context become children of that span.
3. End the span: call span.End() before the function exits, because OpenTelemetry only exports completed spans.

Define span names around operations that map to SLOs and failure modes, add attributes for dimensions used in filtering and grouping such as customer tier, route, or error code, and record events for point-in-time facts such as retries or cache misses. For agentic applications, the same pattern can carry prompts, responses, and context that allows tracing of remote Google Cloud MCP server calls.

3.3Choosing a Collection Architecture for Processing and Cost Control

The collection architecture (opens in a new tab) controls where telemetry is processed, filtered, and exported to a Google Cloud project, which determines vendor coupling and where cost-saving logic can run. The OpenTelemetry Collector configuration file controls how the collector receives, processes, and exports telemetry, and Google Cloud recommends OTLP for both receiving from applications and exporting through the Telemetry (OTLP) API. Collector-based pipelines keep application code vendor-neutral and centralize processing, while direct export removes the collector operational burden but embeds vendor-specific export dependencies in the application.

Traffic flows from the instrumented application to the collector and then from the collector to the Google Cloud project in the collector model, while traffic flows directly from the application to the Google Cloud project in the direct model. Use a collector when the compute environment supports one and when tail sampling, attribute redaction, batching, or per-signal routing is needed to reduce ingestion volume. Use direct in-process export when running a separate collector is impractical, such as constrained Cloud Run or function lifecycles, and accept that sampling and filtering must then be decided in code.

On Google Kubernetes Engine (GKE), Managed OpenTelemetry for GKE provides managed collection through an in-cluster OTLP endpoint without operating a collector, plus optional automatic configuration through the instrumentations.telemetry.googleapis.com Instrumentation custom resource. Managed collection suits teams that want correlated traces, metrics, and logs with minimal operations, while the self-deployed Google-built collector suits teams that need collector-level filtering and controls. On Compute Engine, the Ops Agent with an OTLP receiver collects OTLP metrics and traces from the SDK, transforms them, and sends metrics through the Cloud Monitoring API and traces through the Cloud Trace API.

3.4Managing Aggregation, Quotas, and Ingestion Cost Against SLOs

Aggregation, quotas, and ingestion pricing together set the sustainable operating point for tracing under SLO pressure. Cloud Trace aggregates ingested spans into per-request latency views and service-wide latency distributions, so fewer but richer sampled traces can still reveal bottlenecks, diagnose failures, and verify latency objectives. When telemetry is sent to Google Cloud, billing is by ingestion volume, with traces billed under Cloud Trace pricing, metrics under Google Cloud Managed Service for Prometheus pricing, and logs under Cloud Logging pricing.

The service depends on quota boundaries that sit between the pipeline and storage: Cloud Logging quotas and limits, Cloud Monitoring quotas and limits, and Cloud Trace quotas and limits all apply, including to Managed OpenTelemetry for GKE traffic. Design retention and query strategy around those constraints by sampling aggressively for high-volume success paths, preserving detailed custom spans and attributes for SLO-critical and error paths, and using collector processors to drop, aggregate, or route telemetry before ingestion charges and quota consumption occur. This keeps end-to-end request flows observable enough to identify bottlenecks and optimize reliability without paying to store repetitive spans that do not change SLO decisions.

Key Facts

• OpenTelemetry spans are created by a Tracer with an instrumentation scope name and only completed spans that call End() are exported to Cloud Trace.
• Collector-based export keeps application code vendor-neutral by holding Google Cloud configuration in the collector, while direct export sends OTLP directly to the Google Cloud project through the Telemetry API without a collector.
• The W3C traceparent header propagates trace context across services, and missing child spans or split traces indicate failed propagation.
• Cloud Run standard sampling captures at most 0.1 requests per second per instance and forced tracing raises the limit to about 10 requests per second per instance.
• The roles/cloudtrace.user role permits reading traces, the roles/cloudtrace.agent role permits writing traces, and the roles/cloudtrace.admin role permits both.