A financial services organization runs a high-throughput microservices application on Google Kubernetes Engine (GKE) processing over 50,000 requests per second. The DevOps engineering team is experiencing excessive Google Cloud Trace ingestion costs while attempting to monitor Service Level Objectives (SLOs). The team has established the following operational requirements: Error capture: 100% of failed transactions (HTTP 5xx responses and gRPC error statuses) must be retained for root-cause analysis. SLO performance monitoring: All requests exceeding the 99th percentile latency threshold (1,500 ms) must be captured. Cost governance: High-volume, healthy, and low-latency transactions must be limited to a 1% baseline capture rate to remain within GCP budget constraints. Which OpenTelemetry tracing and sampling strategy should the team implement?

Deploy the OpenTelemetry Collector in a gateway topology and configure the tail_sampling processor with policies for status code, latency threshold, and probabilistic rate limiting.: ### What the solution is The OpenTelemetry Collector **tail_sampling processor** evaluates sampling decisions at the end of a trace lifecycle rather than at the beginning. By deploying the OpenTelemetry Collector in a gateway (cluster-level) architecture, spans from across all downstream microservices are routed and assembled into complete trace graphs before determining whether to export them to **Cloud Trace** or discard them. ### How it meets the requirements - **Error capture:** The collector evaluates the final trace status code, applying a `status_code` filter (e.g., `ERROR`, `STATUS_CODE_ERROR`) to retain 100% of spans associated with failed requests. - **SLO performance monitoring:** The `latency` filter policy inspects the total trace duration and automatically retains traces exceeding the 1,500 ms threshold for end-to-end latency analysis. - **Cost governance:** Traces that do not match error or high-latency criteria fall through to a `probabilistic` policy configured at 1% (`0.01`), effectively dropping 99% of routine telemetry before transmission to Google Cloud Trace. ### Key advantages and capabilities - **Holistic trace visibility:** Unlike head-based samplers, tail-based sampling makes decisions based on the outcome and duration of the entire distributed transaction across multiple services. - **GCP cost reduction:** Prevents unneeded span data from reaching the **Cloud Trace API**, substantially reducing ingestion billing. - **Configurable policy chaining:** Supports composite evaluation matrices (latency, error status, HTTP route, and custom attributes) within a single pipeline. - **Centralized governance:** Centralizes telemetry filtering logic within the collector pipeline without modifying microservice application source code. ### Why this is optimal Head-based sampling occurs at span initialization before the outcome or duration is known. Tail sampling in the OpenTelemetry Collector gateway is the industry standard for enforcing fine-grained, outcome-based filtering rules that balance diagnostic depth against Cloud Trace ingestion quotas.

Opentelemetry Tail Sampling Cloud Trace Cost Slo | Professional Cloud DevOps Engineer

Opentelemetry Tail Sampling Cloud Trace Cost Slo

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A financial services organization runs a high-throughput microservices application on Google Kubernetes Engine (GKE) processing over 50,000 requests per second. The DevOps engineering team is experiencing excessive Google Cloud Trace ingestion costs while attempting to monitor Service Level Objectives (SLOs).

The team has established the following operational requirements:

Error capture: 100% of failed transactions (HTTP 5xx responses and gRPC error statuses) must be retained for root-cause analysis.
SLO performance monitoring: All requests exceeding the 99th percentile latency threshold (1,500 ms) must be captured.
Cost governance: High-volume, healthy, and low-latency transactions must be limited to a 1% baseline capture rate to remain within GCP budget constraints.

Which OpenTelemetry tracing and sampling strategy should the team implement?

Configure head-based TraceIdRatioBased sampling at 1% across all microservice OpenTelemetry SDKs, and configure applications to override parent spans to 100% when an uncaught exception is thrown.

Configure the OpenTelemetry spanmetrics processor to convert 100% of spans into Cloud Monitoring metrics, then drop all trace spans at the collector export layer.

Deploy the OpenTelemetry Collector in a gateway topology and configure the tail_sampling processor with policies for status code, latency threshold, and probabilistic rate limiting.

Export 100% of spans directly to Google Cloud Trace, and create Cloud Logging exclusion filters with a 99% drop rate on healthy trace spans.

Professional Cloud DevOps Engineer

Opentelemetry Tail Sampling Cloud Trace Cost Slo

Related questions