Instrumenting Code with Metrics Logs and Traces

Cloud Trace (opens in a new tab) is a distributed tracing system for Google Cloud that collects latency data and shows it in near real time in the Google Cloud console. A trace follows one request across microservices, and each trace is made of spans where each span records one operation inside that request. By joining all spans for a request, Cloud Trace shows request propagation, service dependencies, and where time was spent.

1.1How Cloud Trace Visualizes Request Propagation

Cloud Trace visualization shows the path of a request from service to service and how long each step takes. In Trace Explorer, a trace appears as a Gantt chart where each span shows start time, duration, attributes, and metadata. Selecting a span reveals which service or database operation added latency to the overall request.

The same trace data also supports service views that show traffic flow between services. Cloud Service Mesh (opens in a new tab) builds a service topology graph from sidecar proxy telemetry, where each sidecar proxy intercepts inbound and outbound HTTP traffic and reports latency, traffic, and errors. Traffic therefore flows from the upstream service through its sidecar proxy to the downstream service, and operators can see slow edges without manually stitching spans together.

1.2Manual Spans and Trace Context Propagation

Manual span instrumentation is the practice of adding code that starts a span, adds attributes, and ends the span around important operations. Instrumented code creates a trace ID for the whole request and a span ID for each operation, then forwards that trace context from the upstream service to each downstream service. When every service forwards the context it received, Cloud Trace can join all spans into one end-to-end trace.

Sampling controls what fraction of requests produce traces, which balances visibility with cost and volume. In gRPC observability configuration, the cloud_trace.sampling_rate field sets this fraction, so a value of 0.5 randomly traces 50% of remote procedure calls (RPCs). If a service does not propagate trace context downstream, the chain breaks and Cloud Trace shows disconnected fragments instead of one complete path.

1.3Correlating Traces with Logs for Troubleshooting

Correlated telemetry means logs, traces, and metrics share the same request identity so they can be investigated together. When tracing is enabled, log entries sent to Cloud Logging (opens in a new tab) during a request contain trace, spanId, and traceSampled fields that link a log line to its exact span and trace. This linkage helps explain slow spans and errors with surrounding events.

Envoy access logs enabled through Istio configuration provide another source of correlated data for mesh traffic. The information in Google Cloud Observability access logs originates from Envoy access logs and includes headers for routing and origin:

• route_name identifies the route that handled the request
• upstream_cluster identifies the destination cluster for the request
• X-Envoy-Original-Path preserves the original request path
• X-Envoy-Original-Host preserves the original request host

Correlated logs support practical troubleshooting workflows. Traffic logs can be exported to BigQuery to query slow requests, such as requests taking more than 5 seconds, and can be used to create log-based metrics and to troubleshoot 404 and 503 errors by connecting status, latency, route, and upstream details to the trace.

Structured logging and error reporting is the practice of writing application logs as structured JSON with consistent severity and metadata so failures can be searched and grouped automatically. Cloud Logging (opens in a new tab) stores, searches, analyzes, monitors, and alerts on log data and events, while Error Reporting aggregates errors from running applications into groups. Together they let developers move from a single error to its request context, related logs, and trace.

2.1Structured Logging with Cloud Logging Client Libraries

Cloud Logging client libraries for Node.js provide the code path for emitting structured logs from application code. The libraries include Bunyan and Winston plugins and a Stackdriver Logging API client library, so teams can keep their existing logger while sending output to Cloud Logging. Log data flows from the instrumented application to Cloud Logging, where entries are stored as JSON objects with named fields instead of plain text. This structure preserves rich metadata that later supports filtering, alerting, and correlation.

2.2Severity Levels and Rich Metadata for Log Explorer

Severity and metadata in Cloud Logging determine how an entry can be found and understood in Log Explorer. Each structured entry carries a severity field such as INFO, plus a timestamp, logName, resource, httpRequest, and labels. The resource object identifies where the log originated, for example a k8s_container with cluster_name, namespace_name, pod_name, container_name, project_id, and location. The httpRequest object captures request details such as requestMethod, requestUrl, status, latency, remoteIp, and protocol, while labels carry service-aware attributes such as source and destination workload, namespace, canonical service, route_name, and upstream_cluster.

Structured fields make troubleshooting efficient because Log Explorer can filter on severity, service, request, and resource labels. The same log data also supports operational workflows:

• Integrate with Cloud Trace (opens in a new tab) to link requests to latency data
• Export traffic logs to BigQuery to query slow requests, such as requests taking more than 5 seconds
• Create log-based metrics from structured fields
• Troubleshoot 404 and 503 errors using request and Envoy-derived headers

Once stored, structured entries can therefore be searched by what happened, where it happened, and which request it belonged to.

2.3Correlating Logs with Traces Through Context Propagation

Log-to-trace correlation connects a log entry to the distributed request that produced it. When tracing is enabled, Cloud Logging entries include trace context fields such as trace, spanId, and traceSampled. Data flows from the trace system into the log entry, so one request identifier links logs across services that handled the same request. The example access log shows a full trace name in trace, a hex identifier in spanId, and traceSampled set to true, which together allow the entry to be opened alongside its trace.

2.4Automated Error Grouping with Error Reporting

Error Reporting is the service that aggregates repeated errors from running applications into groups for diagnosis. Error Reporting consumes error data reported by instrumented code and structured logs, so identical failures are grouped rather than shown as thousands of separate entries. Developers use Cloud Logging to store and search the detailed log context and use Error Reporting to see frequency, affected services, and representative samples. This division keeps investigation service-aware: Cloud Logging answers what happened in a specific request, and Error Reporting answers how often the same failure occurs across the application.

Custom metrics are numeric time-series measurements that you define in application code to track domain-specific performance indicators. Unlike system metrics such as CPU utilization, memory usage, and disk usage that Google Cloud services collect automatically, custom metrics show what is happening inside application logic, frameworks, or language runtimes. Custom metrics are stored in your Google Cloud project and become queryable time series that power charts in Metrics Explorer, custom dashboards, and alerting policies in Cloud Monitoring (opens in a new tab).

3.1Why Custom Metrics Are Needed for Troubleshooting

Custom metrics fill the gap left by system metrics when diagnosing application-level problems. System metrics can show that a problem exists, but they do not explain application behavior such as latency for a specific function, queue depth, checkout failures, or token usage. Instrumentation code that runs inside the application process collects this domain-specific information and exports it as telemetry to a configured storage system. In Google Cloud, that storage system is typically your Google Cloud project, where Google Cloud Observability services collect, analyze, and correlate metric data with log data and trace data.

System metrics flow automatically from the Google Cloud service to Cloud Monitoring, while custom metrics depend on code you add. The application depends on the instrumentation framework to generate telemetry, and the framework depends on an exporter to send that telemetry to the project. Once stored, the same metric time series can be charted over time to establish baselines under different load conditions, compared across versions, and combined with logs and traces to find root cause.

3.2Developing Custom Metrics with OpenTelemetry and Client Libraries

Developing custom metrics means adding instrumentation that measures application behavior and attaches meaningful context. OpenTelemetry (opens in a new tab) is the recommended vendor-neutral open source framework for this work, and you can also instrument with Cloud Client Libraries or the Cloud Monitoring API directly. OpenTelemetry provides a standardized data model and interoperable libraries for traces and metrics, so the same approach works across supported languages and you can send data to Google Cloud or to other vendors without rewriting code.

Vendor-neutral instrumentation provides three related benefits:

• No vendor lock-in: the framework owns its data model, so you can send telemetry to multiple vendors and change destinations without modifying application code
• Standardized procedures: the same framework collects telemetry the same way across services, so metric data from different services can be collected and compared
• Interoperable libraries: trace libraries and metric libraries work together, so you can use either signal or both signals from one framework

Choose OpenTelemetry when you need portable, service-aware diagnostics with coordinated metrics, logs, and traces. Choose direct calls to the Cloud Monitoring API or Cloud Trace API when you need a vendor-specific path without a framework. For databases, the Sqlcommenter OpenTelemetry library automatically instruments object-relational mappings (ORMs) to augment SQL statements with tags and propagate trace context to the database, which correlates application code with database performance.

3.3Exporting Metrics to a Google Cloud Project

Exporting metrics moves telemetry from the application process to metric storage in your Google Cloud project. The instrumentation code lives inside the application process and interacts with the application to generate telemetry, then an exporter sends that telemetry to the destination. The recommended pattern uses an OpenTelemetry Protocol (OTLP) exporter in the application that sends metric data to a collector, and the collector forwards the data to the project through the Telemetry API.

Export follows an ordered flow:

1. Generate: in-process instrumentation measures the domain indicator and creates metric data points with labels and resource context
2. Export: an OTLP exporter, Cloud Monitoring exporter, Ops Agent receiver, or platform sidecar sends the data from the workload to the Google Cloud project
3. Store and scope: Cloud Monitoring stores the time series in the project, and the metrics scope of the project determines which stored time series can be charted or monitored

The export path depends on where the application runs. On Compute Engine, install the Ops Agent to collect host metrics and to receive Prometheus and OTLP data. On Google Kubernetes Engine (GKE), Google Cloud Managed Service for Prometheus handles third-party and user-defined metric data. On Cloud Run, use a sidecar that runs alongside the service to forward custom metrics to Cloud Monitoring.

3.4Querying Metrics for Dashboards and Alerts

Monitoring queries turn stored custom metrics into answers for dashboards, exploration, and alerting. A query selects one or more metric time series in a Google Cloud project, filters by labels such as service, version, or customer ID, and aggregates over time to show throughput, latency, error rate, or resource needs. In Cloud Monitoring, those queries power charts in Metrics Explorer, widgets on custom dashboards, and conditions in alerting policies that notify you when monitored data meets criteria you define.

Custom dashboards let you review related signals together, such as performance metric data, log data from Cloud Logging (opens in a new tab), trace latency from Cloud Trace (opens in a new tab), and open incidents. Sophisticated queries written with Monitoring Query Language (MQL) support this workflow by combining, filtering, and aligning custom application metrics with system metrics. Because Google Cloud Observability correlates the signals, a spike in a custom metric such as payment failures can lead directly to related logs and to traces that show how the request propagated across services.

Implementing log and metric instrumentation strategies for reliable diagnostics means aligning business goals to measurable technical signals and instrumenting code so Cloud Monitoring (opens in a new tab) and Cloud Logging (opens in a new tab) emit consistent, high-fidelity data. Each business key performance indicator (KPI), such as revenue from AI recommendations or user engagement from generative AI features, maps to technical reliability metrics like request success rate, response latency, harmful-output rate, and task-completion rate. Those metrics are then quantified as service-level objectives (SLOs) from the user's perspective, so monitoring and early anomaly detection focus on behavior that affects users and the business.

4.1Define Reliability Goals and SLO Targets

Reliability goals in Google Cloud Observability translate KPIs into SLO targets that can be measured with metrics and logs. For a conversational assistant, the corresponding metrics include success rate of user requests, latency as time to first token (TTFT) and token streaming, rate of irrelevant or harmful responses, and rate of successful task completion. For training workloads, reliability metrics include model FLOPS utilization (MFU), iterations per second, tokens per second, and tokens per device. Example SLO targets are 99.9% of API calls must return a successful response, 95th percentile inference latency must be below 300 ms, TTFT must be below 500 ms for 99% of requests, and rate of harmful output must be below 0.1%.

4.2Track Infrastructure Health and the Four Golden Signals

Infrastructure and application performance monitoring tracks resource metrics across all resources used by the system, including processor usage for CPU, GPU, and TPU, memory usage, network throughput and latency, and disk I/O. Coverage spans managed environments like Gemini Enterprise Agent Platform training and serving and self-managed resources like Google Kubernetes Engine (GKE) nodes and Cloud Run instances. Application health is tracked with the four golden signals:

• Latency: time to respond to requests, including inference latency and TTFT
• Traffic: volume of requests or workload handled by the service
• Error rate: rate of failed requests or operations
• Saturation: utilization of critical resources like CPU, memory, and GPU or TPU accelerators, which shows how close the system is to capacity limits

Observing changes in latency, error rates, and throughput during load, failure simulation, or recovery reveals degradation and verifies whether rerouting, failover, and alerting work as intended.

4.3Implement Metric Instrumentation for Continuous Monitoring

Metric instrumentation in Cloud Monitoring collects, stores, and visualizes infrastructure and application metrics for dashboards, charts, and alerts. Teams use pre-built dashboards for Google Cloud services and create custom dashboards tailored to workload-specific indicators and infrastructure health. Metrics from specialized serving frameworks like vLLM or NVIDIA Triton Inference Server flow into Cloud Monitoring through Google Cloud Managed Service for Prometheus, while Agent Platform exports its metrics directly to Cloud Monitoring. Detailed accelerator signals, such as GPU utilization per process, memory usage per process, and temperature, flow from tools like NVIDIA Data Center GPU Manager (DCGM) into the same monitoring path.

For gRPC services, metrics are enabled by adding a cloud_monitoring object to the gRPC observability configuration file, with its value set to {}. Once enabled, the service depends on the configuration file location exported as GRPC_GCP_OBSERVABILITY_CONFIG_FILE to send metrics to the configured Google Cloud project. Alerts on these metrics then trigger for key events such as service outages or failovers, which supports consistent performance monitoring and early detection of saturation and errors.

4.4Implement Log Instrumentation for Troubleshooting Context

Log instrumentation in Cloud Logging collects detailed logs from applications and underlying infrastructure to provide context around events and errors. Cloud Logging stores, searches, analyzes, monitors, and alerts on log data and events, and application libraries such as Bunyan and Winston plugins forward structured entries to the service. Each high-fidelity entry carries httpRequest fields like request method, URL, status, sizes, and latency, plus resource fields like cluster, namespace, pod, and container, plus severity such as INFO, plus timestamp and labels for source, destination, route, and authentication policy. When tracing is enabled, the same entry also carries trace, spanId, and traceSampled, so a log line points directly to its distributed trace.

Log data supports diagnostics beyond search, because traffic logs can be exported to BigQuery for queries such as selecting all requests that take more than 5 seconds, converted into log-based metrics for dashboards and alerts, and used to troubleshoot 404 and 503 errors. The three core observability signals divide by diagnostic role:

Cloud Profiler (opens in a new tab) complements these signals by identifying bottlenecks within function blocks in application code, which helps optimize resource usage and execution time.

4.5Set Sampling Strategies and Event Boundaries

Sampling strategies and event boundaries control telemetry volume while preserving diagnostic fidelity. In the gRPC observability configuration, the cloud_trace object sets sampling_rate to a fraction of remote procedure calls (RPCs) to trace, and the cloud_logging object sets client_rpc_events and server_rpc_events patterns that define which services or methods generate transport-level event logs. Each pattern names methods such as google.pubsub.v1.Subscriber/* or *, sets exclude to true to skip noisy methods like acknowledge or create-topic calls, and sets max_metadata_bytes and max_message_bytes such as 4096 to bound how many header and message bytes are logged. Labels such as SERVICE_NAME, SOURCE_VERSION, and DATA_CENTER attach service-aware context to every signal, so filtered telemetry still identifies which service and deployment produced it.

Choosing narrow method patterns with byte limits keeps high-traffic paths affordable, while broader patterns on critical paths preserve the detail needed to detect anomalies early without overwhelming storage and analysis.

Context propagation and signal correlation is the practice of carrying request identity across services so metrics, logs, and traces for the same work can be joined together. In multi-service applications on Google Cloud, an upstream service starts or receives trace context and passes it downstream, while each service adds service-aware labels and structured fields. Google Cloud Observability then joins the signals in Cloud Monitoring, Cloud Logging (opens in a new tab), and Cloud Trace (opens in a new tab), so a slow request, its log lines, and its latency breakdown share the same identifiers. That shared context turns isolated signals into an end-to-end story for root-cause analysis.

5.1Propagating Context Across Service Boundaries

Propagating context across service boundaries means each service forwards trace identity, request metadata, and application tags to the next hop in the request path. Instrumentation is typically done with OpenTelemetry, using collector-based export for Go, Java, Node.js, and Python or zero-code instrumentation for Java workloads on Google Kubernetes Engine (GKE). For gRPC services, trace context received from upstream or started locally must be propagated downstream, or traces will break at the service boundary and downstream spans will appear disconnected. Application context can also travel out of band, such as sqlcommenter adding Model-View-Controller framework information to SQL queries as a comment so database activity maps back to application code. Custom service labels such as SERVICE_NAME, SOURCE_VERSION, and DATA_CENTER travel with telemetry to identify which revision and location produced each signal.

Sampling and export settings control how much context flows and where the telemetry lands. In gRPC observability configuration, cloud_trace.sampling_rate set to 0.5 randomly traces 50% of remote procedure calls, which balances visibility against cost and volume. Applications that still use direct export with the Trace exporter migrate to sending OpenTelemetry Protocol (OTLP)-formatted trace data to the OTLP endpoint. The instrumentation approach is therefore chosen by platform and need: collector-based code changes for full control, zero-code for Java on GKE when code cannot change, and consistent propagation checks wherever one service calls another.

5.2Associating Logs with Traces Using Correlation Identifiers

Associating logs with traces means writing the same trace ID, span ID, and sampling flag into every log entry for a request. When tracing is enabled, Cloud Logging entries carry trace information that Cloud Trace uses to link a log line to a specific span in a distributed trace. The access logs produced by Envoy in an Istio service mesh show this pattern, where network metadata, service identity, and trace correlation travel together from the proxy to Cloud Logging.

The correlation identifiers that join a log entry to its trace include:

• trace: full resource name of the trace, such as projects/PROJECT_ID/traces/TRACE_ID, which links the log entry to the trace in Cloud Trace
• spanId: identifier of the individual operation span that generated the log, which places the log at a point in the Gantt chart
• traceSampled: boolean flag that records whether the trace was sampled and retained for analysis
• request_id: mesh request identifier that stays constant as the request moves from ingress gateway to destination workload
• Service-aware labels: source and destination canonical service, namespace, workload, version, and revision, which identify both sides of each hop

Traffic flows from the ingress gateway to the destination workload, and the log entry records both sides through labels such as source_canonical_service and destination_canonical_service. Because the log carries trace, spanId, and traceSampled together, selecting the log in Cloud Logging can open the matching trace, and selecting a span in Cloud Trace can show the logs emitted during that span. Log entries that the application sends to Cloud Logging during a sampled database request are linked the same way, so an end-to-end application trace displays database activity alongside application logs for one investigation.

5.3Correlating Metrics, Logs, and Traces for Root-Cause Analysis

Correlating metrics, logs, and traces for root-cause analysis means starting from a service health signal and drilling into the linked trace and logs for the same request. Cloud Monitoring collects infrastructure and application metrics and visualizes them with pre-built and custom dashboards, charts, and alerts, while Cloud Logging stores, searches, and analyzes event context and Cloud Trace collects latency data across services in near real time. Error Reporting aggregates errors with recognized stack traces, and Cloud Profiler (opens in a new tab) identifies bottlenecks inside function blocks so teams can optimize resource usage and execution time. Together the services depend on shared context: Monitoring detects the symptom, Trace shows which service hop added latency, and Logging shows what the code was doing at that moment.

Monitoring for AI, ML, and microservice workloads focuses on the four golden signals:

• Latency: time to respond to requests, including time to first token and token streaming for large language models
• Traffic: volume of requests or workload handled by the service
• Error rate: rate of failed requests or operations, including irrelevant or harmful model outputs
• Saturation: utilization of critical resources such as CPU, memory, GPU, or Tensor Processing Unit accelerators, which shows how close the system is to capacity limits

Service-level objectives make those signals actionable, with targets such as 99.9% successful API calls or 95th percentile inference latency below 300 ms. When latency, error rate, or saturation breaches a target, the alert points to the service and time window, then the trace for a slow sample query shows each span with attributes and metadata in a Gantt chart. The in-context trace can be opened in Trace Explorer with View in Cloud Trace, where the linked log entries explain errors, retries, and downstream calls that metrics alone cannot show.

Observability Analytics extends correlation by querying log views and the _AllSpans trace view with SQL to filter, group, and join telemetry across services. A query can extract payload fields, calculate latency percentiles, and join log entries to spans on the shared trace identifier to find requests that took more than 5 seconds or returned 404 and 503 errors. Traffic logs exported to BigQuery support the same joins at larger scale, and log-based metrics turn repeated correlated events into new Monitoring signals. In this flow, context moves from code to collector or OTLP endpoint to Observability backends, and analysis moves from metric to trace to log without losing the request identity that binds them.

Key Facts

• Cloud Trace joins spans that share a trace ID into one end-to-end trace, but the chain breaks if any service fails to propagate trace context downstream.
• A log entry links to Cloud Trace only when it carries trace, spanId, and traceSampled together.
• A gRPC cloud_trace.sampling_rate of 0.5 randomly traces 50% of remote procedure calls (RPCs) and does not control log volume.
• Custom metrics track domain-specific application behavior and power Metrics Explorer charts, custom dashboards, and Cloud Monitoring alerting policies, while system metrics are collected automatically.
• On Cloud Run, the Prometheus sidecar forwards Prometheus metrics and the OpenTelemetry sidecar forwards OTLP metrics to Cloud Monitoring.