A production microservices application running on Google Kubernetes Engine (GKE) is experiencing severe tail latency during peak checkout periods. A DevOps engineer investigates the issue across Google Cloud Observability tools and observes the following telemetry: Cloud Monitoring: The GKE nodes and pods report low average CPU utilization (under 25%) and normal memory usage, but p99 latency alerts are firing consistently. Cloud Trace: Distributed request traces reveal that the checkout service spends over 3.2 seconds waiting on a specific internal span inside the payment-auth service, while downstream database calls within that span complete in less than 15 milliseconds. Cloud Profiler: The payment-auth service exhibits very low CPU time, but shows extremely high Wall time and Contention concentrated on a shared synchronization mutex routine in the application runtime. Which root cause and diagnostic conclusion do these correlated telemetry signals indicate?

The bottleneck is caused by application-level thread or lock contention inside the service, where threads are blocked waiting for shared resources rather than consuming CPU cycles or waiting on downstream databases.: ### What Thread and Lock Contention Is **Thread contention** or **lock contention** occurs when multiple concurrent threads attempt to acquire an exclusive lock (such as a synchronized block, mutex, or semaphore) simultaneously. When one thread holds the lock, all other competing threads are placed into a blocked or waiting state, stalling execution without executing active instructions on the host CPU. ### How It Correlates Across Observability Telemetry Correlating data across **Google Cloud Observability** tools pinpoints this exact pathology: - **Cloud Monitoring** shows low **CPU utilization** because blocked threads yield their execution state to the operating system scheduler rather than spinning on CPU cores. - **Cloud Trace** breaks down the end-to-end request lifecycle into discrete **spans**, proving that downstream dependencies (such as databases or external APIs) respond rapidly (15 ms) while the enclosing local service span accounts for over 3 seconds of wall-clock delay. - **Cloud Profiler** captures execution characteristics by distinguishing **CPU time** (active on-core cycles) from **Wall time** (total elapsed clock time) and **Contention** (time spent waiting on locks and mutexes). High wall and contention metrics with low CPU metrics definitively isolate the synchronization lock in the application code. ### Key Advantages of Correlated Observability - **Eliminates unnecessary infrastructure scaling**: Prevents costly and ineffective horizontal pod autoscaling that would not solve serialized thread lock queues. - **Pinpoints exact code paths**: Maps slow latency directly to the offending function call or lock boundary in source code. - **Differentiates infrastructure vs. code defects**: Separates downstream external bottlenecks from in-process synchronization barriers. - **Accelerates root-cause analysis (RCA)**: Minimizes MTTR by systematically eliminating infrastructure starvation and database degradation. ### Why This Is Optimal Recognizing that low CPU utilization paired with high wall/contention time reflects code-level locking allows developers to optimize data structures, reduce synchronization scope, or adopt non-blocking asynchronous patterns rather than misattributing latency to GKE capacity or database throughput.

Correlate Telemetry Diagnose Contention Bottleneck | Professional Cloud DevOps Engineer

Correlate Telemetry Diagnose Contention Bottleneck

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A production microservices application running on Google Kubernetes Engine (GKE) is experiencing severe tail latency during peak checkout periods. A DevOps engineer investigates the issue across Google Cloud Observability tools and observes the following telemetry:

Cloud Monitoring: The GKE nodes and pods report low average CPU utilization (under 25%) and normal memory usage, but p99 latency alerts are firing consistently.
Cloud Trace: Distributed request traces reveal that the checkout service spends over 3.2 seconds waiting on a specific internal span inside the payment-auth service, while downstream database calls within that span complete in less than 15 milliseconds.
Cloud Profiler: The payment-auth service exhibits very low CPU time, but shows extremely high Wall time and Contention concentrated on a shared synchronization mutex routine in the application runtime.

Which root cause and diagnostic conclusion do these correlated telemetry signals indicate?

The bottleneck is caused by inter-service network packet drop and VPC latency between GKE pods, which requires enabling cross-zone network peering and optimizing MTU settings.

The bottleneck is caused by compute resource starvation on the GKE cluster, requiring a vertical increase in GKE node CPU machine types and pod CPU limits.

The bottleneck is caused by database resource exhaustion and high query latency, which requires increasing database compute capacity and adding secondary indexes.

The bottleneck is caused by application-level thread or lock contention inside the service, where threads are blocked waiting for shared resources rather than consuming CPU cycles or waiting on downstream databases.

Professional Cloud DevOps Engineer

Correlate Telemetry Diagnose Contention Bottleneck

Related questions