Prepare and test your skills
Prepare and test your skills
Application instrumentation and span management is the practice of adding tracing code so each request produces a trace made of timed spans. A trace is a group of spans that share one unique trace ID, while each span measures one discrete operation such as a database query, an API call, or business logic. In Google Cloud, Cloud Trace (opens in a new tab) stores these traces and shows them as waterfall graphs so teams can see end-to-end latency and find slow layers.
Automatic tracing in Cloud Run (opens in a new tab) gives a baseline measurement without code changes. Incoming requests to Cloud Run automatically generate traces, and the standard W3C traceparent header is populated so the trace ID can be passed onward. Automatic traces show only the outer request, not what happens inside each layer. Add instrumentation when granular insight is required, such as timing a call to the Cloud Translation API or showing request flow across multiple services as a single trace.
Cloud Run samples automatic traces at a maximum rate of 0.1 requests per second for each instance, or about one request every 10 seconds, and the sample rate cannot be configured. A specific request can be forced to be traced, which raises the limit to about 10 requests per second for each instance. Automatically generated Cloud Run traces do not result in billing charges, while custom spans added with libraries and correlated to Cloud Run spans incur standard Cloud Trace charges.
Trace data flow is the ordered path that carries spans from instrumented code to a Google Cloud project for storage and analysis. The recommended approach uses vendor-neutral OpenTelemetry with an OTLP exporter that sends OpenTelemetry Protocol (OTLP) data to a collector. The collector runs alongside the application and offloads work from the service by handling retries, batching, and encryption before export to the observability backend through the Telemetry (OTLP) API.
The collector-based path moves data through a fixed sequence:
This order keeps the application fast because delivery work moves from the application to the collector. Other supported paths cover cases where a collector cannot be used or where the workload needs extra collection.
| Export path | How data flows | Use when |
|---|---|---|
| Collector-based OTLP | Application -> Collector via OTLP -> Google Cloud project via Telemetry API | The recommended default for resilience with retries and batching |
| Direct OTLP | Application -> Google Cloud project via Telemetry API | A collector cannot be run alongside the application |
| Ops Agent with OTLP receiver | Application -> Ops Agent on Compute Engine -> Google Cloud project | The workload runs on Compute Engine and also needs logs and metrics collection |
The Ops Agent path has one boundary to remember: the Ops Agent can collect traces and metrics through OTLP, but it does not collect logs through OTLP.
Parent spans and child spans form a hierarchy that shows how one request breaks into nested operations under a single trace ID. The parent span represents the outer operation, such as an incoming Cloud Run request, and child spans represent inner work, such as a database query or downstream API call that runs before the parent completes. All spans in the hierarchy carry the same trace ID, so Cloud Trace can correlate spans from disparate services into one end-to-end view. Correct propagation of trace context from service to service is what keeps the hierarchy intact.
OpenTelemetry and client libraries automatically create spans at remote procedure call (RPC) boundaries, which gives a baseline hierarchy without manual code. Custom spans extend that hierarchy for operations the automatic spans do not isolate, and labels can be added to custom spans to make them easier to filter in the Cloud Trace UI. In the Trace console the result appears as a waterfall graph where each child span sits under its parent in time order, which makes granular latency analysis possible.
Exam tip: Automatic Cloud Run traces are free, while custom spans correlated to those Cloud Run spans are billed at standard Cloud Trace rates.
Client-side spans capture performance as experienced by the calling application, including network and client-system latency that server traces cannot see. Client-side traces are collected when the client executes RPCs and are useful for debugging Firestore in Datastore mode (opens in a new tab) and Firestore (opens in a new tab) requests end to end. Java and Node.js client libraries can produce these spans using OpenTelemetry APIs and exporters.
Client-side spans collected from RPC execution provide:
These details depend on the observability backend for storage, so collecting traces with OpenTelemetry itself has no charge but ingesting spans into Cloud Trace or another backend may be billable. To control cost, start with a small trace sampling ratio that traces only a small percentage of RPCs based on traffic.
Correlation and diagnostic analysis in Cloud Trace (opens in a new tab) is the practice of joining timing measurements from different services into one end-to-end view of a request. A trace is a set of spans that share the same trace ID, where each span measures the time spent in one layer or service. By correlating spans with that ID and with application logs, teams can see cross-service dependencies, find which service adds latency, and spot outliers in the Google Cloud console.
Span correlation with trace IDs links spans from disparate services into a single distributed trace. Context is carried between services in the standard W3C traceparent header, which Cloud Run (opens in a new tab) automatically populates for incoming requests. Without added instrumentation, Cloud Trace shows only the automatically generated Cloud Run request span, so the full path stays invisible. With instrumentation that propagates that context, Cloud Trace shows the request flow across multiple services as a single request with one trace ID.
Propagation depends on both sides sharing the same ID in order. First, the upstream service sends the trace ID in the traceparent header with the outbound call. Then, the downstream instrumented service creates its spans under that same trace ID. Only after both sides share the ID does the console join their spans into one waterfall view, which makes missing propagation appear as broken or separate traces.
Diagnostic spans in Cloud Trace measure how long a request spends inside each layer of an implementation. Instrumentation libraries automatically create spans at remote procedure call (RPC) boundaries, and custom spans can be created for code that would otherwise be invisible. Typical layer measurements include:
These spans flow from the application through an OpenTelemetry exporter to the Google Cloud project. The recommended path sends OpenTelemetry Protocol (OTLP) data from the application to a collector, and the collector sends the data to the project by using the Telemetry (OTLP) API. The collector runs alongside the application and handles retries, batching, and encryption, which lets the application offload data quickly. Adding labels to custom spans makes it easier to filter for those spans in the Cloud Trace user interface.
A waterfall graph in Cloud Trace is the visual form of a correlated trace, where each span is a bar whose length reflects its latency values. The graph shows parent and child relationships, so a long child span identifies the specific service bottleneck while gaps between spans reveal network or queuing delay. Comparing the same graph across many requests exposes latency outliers and repeated cross-service dependencies. Client-side traces add the performance as experienced by the application, including latency introduced by the network and client system.
Each client-side span carries diagnostic detail beyond start and end timestamps:
Correlating these attributes and event logs with the span timing connects what the code was doing to how long it took. For Firestore, this client-side detail is available for the Java and Node.js client libraries in Native mode and for the Java client library in Datastore mode (opens in a new tab), with detail for Native mode (opens in a new tab) in the same libraries, but no spans are produced for real-time listeners. Starting with a small sampling ratio limits ingestion volume while teams learn normal latency patterns.
Exam tip: Automatic Cloud Run traces show single-service latency for free, but showing one request across multiple services as a single trace requires instrumentation that propagates trace context.
Sampling in Cloud Trace controls which requests produce stored traces. Cloud Run does not sample every request and does not support configuration of its sample rate. Requests are sampled at a maximum rate of 0.1 requests per second for each instance, or about one request every 10 seconds. A particular request can be forced to be traced, which raises sampling to up to 10 requests per second for each instance for diagnostic capture.
| Trace type | What it captures | Billing |
|---|---|---|
| Automatic Cloud Run trace | Incoming request latency without code changes | No charge, whether sampled or forced |
| Custom instrumented spans correlated to Cloud Run spans | Layer timing and multi-service flow under one trace ID | Charged at standard Cloud Trace ingestion rates |
Collecting data with OpenTelemetry itself has no charge, but ingestion of spans into the observability backend may be billable. Because custom spans add cost and automatic traces do not, use automatic traces for baseline latency checks and add custom correlated spans when there is a need to isolate a database, API, or cross-service bottleneck.
Trace context propagation is the process of passing tracing identity from one service to the next so all work for one request shares a unified trace ID. Propagation relies on standard HTTP headers carried with each request, and Cloud Trace (opens in a new tab) groups the spans that share that ID into one end-to-end view. Without propagation, each service creates an isolated trace; with propagation, synchronous calls and asynchronous paths appear as one connected request flow.
Automatic tracing in Cloud Run (opens in a new tab) gives every service a starting trace without code changes. An incoming request to a Cloud Run service automatically generates a trace that can be viewed in Cloud Trace to identify latency sources, and the standard W3C TraceContext traceparent header is automatically populated for Cloud Run requests. Each layer-specific measurement added later is a span, such as the time to complete a database query, receive results from an API request, or run complex business logic. Spans from the same trace appear in Cloud Trace as a waterfall graph that reflects latency values across layers.
Automatically generated traces in Cloud Run, whether sampled or forced, do not result in billing charges. Cloud Run samples at a maximum rate of 0.1 requests per second for each instance, or about one request every 10 seconds, and the sample rate cannot be configured. A forced trace is sampled at a higher limit, but any custom spans added with libraries and correlated to the Cloud Run spans incur standard Cloud Trace charges.
Instrumentation is the added code that extracts, preserves, and forwards trace context across service boundaries. Traces are generated automatically without instrumentation, but instrumentation is required to propagate trace context so Cloud Trace shows the request flow across multiple services as a single request. Instrumentation is also required to create custom trace spans, for example to measure how long a service waits for the Cloud Translation API.
The propagation flow connects the caller to the downstream service in order. First, the upstream service receives an incoming request that carries the trace ID in its propagation header. Then, instrumentation creates child spans for local work and injects the same trace context into the outgoing request to the next service. Only after the downstream service receives that header and joins its spans to the same trace ID does Cloud Trace display the full path as one trace.
Exam tip: traceparent is populated automatically on Cloud Run requests, but showing multiple services as one request requires instrumentation to forward that context on outbound calls.
OpenTelemetry is the recommended vendor-neutral framework for instrumenting applications to collect trace data and send it to Google Cloud. OpenTelemetry automatically creates spans at remote procedure call (RPC) boundaries, and it can also be used to create custom spans for specific operations. The framework supports sampling controls, span limits, collectors, backend-specific exporters, and flexible configuration for sending data to Cloud Trace or another observability backend.
An application can send OpenTelemetry Protocol (OTLP) trace data to its Google Cloud project through several supported paths:
A collector runs alongside the application and lets the service offload data quickly while the collector handles retries, batching, and encryption. The collector receives OTLP messages from the application, processes the messages, and exports them to the observability backend. This separation keeps propagation lightweight in the request path while still preserving the unified trace ID across synchronous calls and asynchronous work such as queued RPCs and background processing.
traceparent header, and Cloud Trace joins spans with the same trace ID into one waterfall graph.traceparent, sampling at most 0.1 requests per second per instance, or up to 10 per second when forced, with no charge.A trace in Cloud Trace is a group of spans that share one unique trace ID, and Cloud Trace joins those spans into one waterfall graph for end-to-end latency analysis. The trace ID is carried between services in the standard W3C traceparent header, with the upstream service sending it on the outbound call and the downstream instrumented service creating its spans under the same ID.
Automatic tracing in Cloud Run gives baseline measurement of the outer request without code changes and automatically populates the traceparent header for onward propagation. Add instrumentation when granular insight is required, such as timing a call to the Cloud Translation API or showing request flow across multiple services as a single trace, because automatic traces alone show only the outer request.
The recommended default sends OpenTelemetry Protocol data from the application OTLP exporter to a collector, and the collector exports the processed traces to the Google Cloud project through the Telemetry API. The collector runs alongside the application and handles retries, batching, and encryption, which keeps the application fast by moving delivery work off the service.
Last updated