An e-commerce checkout service deployed on Google Cloud experiences periodic latency spikes and intermittent transaction failures. While inspecting distributed trace waterfalls in Cloud Trace, a DevOps engineer observes the following behavior across slow requests: The root span duration exceeds 4,500 ms (normal baseline is 250 ms). An intermediate span named InventoryService.reserveStock shows normal execution time (40 ms), but its child span for Cloud Spanner database operations shows long duration (over 3,800 ms). The Spanner span metadata displays standard query execution times of under 15 ms, but contains custom span attributes showing connection wait times exceeding 3,700 ms, accompanied by related application logs with serviceruntime.googleapis.com rate limiting warnings and thread pool exhaustion. Concurrently, Cloud Monitoring indicates that the Cloud Spanner instance smoothed CPU utilization remains low (under 25%). What is the primary root cause indicated by this trace waterfall and span metadata analysis?

Database client connection pool exhaustion and client-side contention delaying request acquisition before reaching Cloud Spanner: ### What Client-Side Database Contention Is **Client-side connection pool exhaustion** occurs when application instances exhaust their configured maximum pool of database client sessions or TCP connections. In distributed tracing, this manifests as high total span latency for database operations even when the remote database itself executes queries swiftly. ### How It Meets the Scenario Findings * **Span metadata discrepancy**: The trace waterfall shows the overall database span taking over 3,800 ms, yet the internal query execution attribute is only 15 ms. The custom attributes pinpoint over 3,700 ms spent waiting to acquire a connection handle from the local client pool. * **Low backend CPU**: Cloud Spanner smoothed CPU utilization is healthy (< 25%), confirming that the database engine is not experiencing query contention or lock waits on the server side. * **Application logs**: Correlated application logs confirming thread wait states and connection starvation align directly with client-side connection pool misconfiguration. ### Key Advantages and Diagnostic Indicators * **Accurate bottleneck identification**: Differentiating between client wait time and remote server execution time prevents unnecessary database scaling. * **Span attribute correlation**: Examining granular attributes (e.g., `db.connection_wait_ms` vs `db.execution_time_ms`) pinpoints whether latency is in-flight or on-node. * **Holistic observability**: Correlating trace waterfalls with server CPU metrics provides conclusive proof of client versus server bottlenecks. ### Why This Is Optimal Evaluating span metadata and comparing individual child span durations against backend resource utilization is the definitive method to isolate client connection pool bottlenecks from true database performance degradation.

Analyze Span Attributes Trace Waterfall Root Cause | Professional Cloud DevOps Engineer

Analyze Span Attributes Trace Waterfall Root Cause

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An e-commerce checkout service deployed on Google Cloud experiences periodic latency spikes and intermittent transaction failures. While inspecting distributed trace waterfalls in Cloud Trace, a DevOps engineer observes the following behavior across slow requests:

The root span duration exceeds 4,500 ms (normal baseline is 250 ms).
An intermediate span named InventoryService.reserveStock shows normal execution time (40 ms), but its child span for Cloud Spanner database operations shows long duration (over 3,800 ms).
The Spanner span metadata displays standard query execution times of under 15 ms, but contains custom span attributes showing connection wait times exceeding 3,700 ms, accompanied by related application logs with serviceruntime.googleapis.com rate limiting warnings and thread pool exhaustion.
Concurrently, Cloud Monitoring indicates that the Cloud Spanner instance smoothed CPU utilization remains low (under 25%).

What is the primary root cause indicated by this trace waterfall and span metadata analysis?

Database client connection pool exhaustion and client-side contention delaying request acquisition before reaching Cloud Spanner

Cloud Trace ingestion rate limits dropping spans and artificially elongating trace waterfall durations

Network packet loss and routing latency in the VPC between the inventory service and the database instance

Cloud Spanner backend storage saturation causing high disk I/O wait times during transaction commits

Professional Cloud DevOps Engineer

Analyze Span Attributes Trace Waterfall Root Cause

Related questions