Prepare and test your skills
Prepare and test your skills
Performance Monitoring and Bottleneck Identification is the practice of collecting live metrics, finding where requests wait, and changing capacity or caching based on data. In Google Cloud, Google Cloud Managed Service for Prometheus (opens in a new tab) scrapes application metrics, Cloud Monitoring (opens in a new tab) visualizes them, and autoscalers and caches act on the results. This observe, identify, and adjust loop protects both latency and throughput.
Observing load with Managed Service for Prometheus and Cloud Monitoring means proving that traffic metrics flow from workloads into visible graphs. Managed Service for Prometheus scrapes metrics from Pods such as a vLLM model server and stores them for analysis. Metrics for vLLM use the format vllm:metric_name, for example vllm:gpu_cache_usage_perc for cache use. After load is added to the endpoint, the graphs should rise, which confirms that scraping is working.
To verify ingestion in Cloud Monitoring, follow this ordered workflow:
vllm:gpu_cache_usage_perc{cluster='CLUSTER_NAME'} to graph cache use in response to load.Data flows from the vLLM Pods to Managed Service for Prometheus and then to Cloud Monitoring. A rising graph after the load script runs proves the pipeline depends on successful scraping.
Identifying bottlenecks with inference queue and cache metrics means watching two related signals for a vLLM server: queued requests and KV cache exhaustion. The KV cache holds active request state, and when it fills, the queue of waiting requests grows noticeably. The Horizontal Pod Autoscaler (HPA) depends on one Prometheus metric to decide when to add replicas, and the choice controls cost versus latency. The HPA target uses averageValue, which must be set experimentally for each model and latency goal.
| Metric | What it measures | Choose when |
|---|---|---|
| num_requests_waiting | Number of requests waiting in the model server queue | Optimizing for throughput and cost, and latency targets are met at maximum throughput |
| gpu_cache_usage_perc | KV cache utilization, which correlates to requests processed per inference cycle | Workloads are latency-sensitive and queue-based scaling is not fast enough |
When the threshold is crossed, scaling follows a fixed order. First, the HPA scales the LeaderWorkerSet/vllm target, for example from 1 toward a maximum of 2 replicas with metric prometheus.googleapis.com|vllm:num_requests_waiting|gauge and averageValue: 5. Then GKE schedules another Pod, which triggers the node pool autoscaler to add a second node before deploying the second replica. Progress can be watched with kubectl get hpa --watch.
Exam tip: Use num_requests_waiting for throughput and cost, while gpu_cache_usage_perc fits latency-sensitive GPU use cases where waiting for a queue would miss requirements.
Verifying cache efficiency for cross-cloud queries means checking whether repeated reads avoid remote fetches. Intelligent caching in Lakehouse (opens in a new tab) automatically stores remote data blocks locally inside Google Cloud. On the first query, Lakehouse fetches blocks from remote object storage such as AWS Amazon S3 over the public internet using TLS and fills the local cache. Later queries for the same blocks read from local Google Cloud storage, which reduces egress fees for repeated patterns.
Effectiveness is verified in BigQuery job statistics in the JobStatistics2 object. Because one query can touch several providers, statistics include a repeated object_storage_stats entry for each provider accessed:
cloud_provider: The provider hosting the object storage, for example AWS or AZURE.cache_bytes_read: Total bytes read from the local Google Cloud cache, avoiding a remote read.object_storage_bytes_read: Total bytes read directly from the remote provider's object storage.High cache_bytes_read relative to object_storage_bytes_read means the local cache is serving the workload and avoiding remote fetches.
Monitoring cache performance with hit rate and throughput charts means tracking whether the cache keeps useful data and adds capacity. Cloud Storage console charts, also viewable in Cloud Monitoring, expose retention, eviction, and serving efficiency at bucket or project level. Together they show whether low hit rates or high evictions explain repeated remote reads and higher latency.
The available charts include:
Network Optimization and Global Traffic Management is the practice of cutting end-to-end latency for dispersed users by choosing the network entry path, edge termination point, and cache location. In Google Cloud, Network Service Tiers (opens in a new tab) control entry into Google's network, global load balancers route through a single anycast IP, and Cloud CDN (Content Delivery Network) (opens in a new tab) serves content from the edge. Together they decide how many long-distance round trips happen before the first byte reaches the user.
Network Service Tiers are the connectivity options that control how traffic moves between the internet and Google Cloud workloads. With Premium Tier, inbound traffic enters Google's private high-performance network at the point of presence (PoP) closest to the sender, travels inside Google's network to the Compute Engine virtual machine (VM), and outbound traffic exits at the PoP closest to the destination. With Standard Tier, traffic travels over transit internet service providers (ISPs) and enters Google's network at a PoP close to the destination region, which adds hops and latency. Choose Premium Tier when serving global users from multiple regions or when using global load balancing, because global load balancing requires Premium Tier and depends on close edge entry.
Global load balancing distributes external traffic to backends in multiple regions through one anycast IP address. Each client is directed to the Google Front Ends (GFEs) closest to where traffic enters Google's network. GFEs are software-defined systems in edge PoPs that terminate Transmission Control Protocol (TCP) or Transport Layer Security (TLS) sessions at the edge and proxy requests to healthy backends with capacity. If a region fills, new connections spill to the next closest region with capacity while existing connections stay, and if a region fails, traffic fails over automatically.
The two main external options differ by traffic type, flow, and scope:
| Option | Traffic flow | Scope |
|---|---|---|
| Global external Application Load Balancer (opens in a new tab) | Terminates TCP or TLS at GFEs at the edge and proxies HTTP(S) traffic to backends, with URL mapping and Secure Sockets Layer (SSL) offloading | Global |
| External passthrough Network Load Balancer | Forwards TCP, User Datagram Protocol (UDP), Encapsulating Security Protocol (ESP), Generic Routing Encapsulation (GRE), and Internet Control Message Protocol (ICMP) packets unchanged to backend VMs using Maglev hashing on source and destination IP, port, and protocol | Regional, with a global preview supporting multi-region backends |
In a test with backends only in us-central1 and a user in Germany, direct access and the regional passthrough option both showed about 110 ms ping and about 230 ms Time To First Byte (TTFB), because each HTTP request needed a full handshake across the ocean. The global external Application Load Balancer showed about 1 ms ping to the nearby GFE and about 123 ms TTFB, because the handshake ended at the edge and only the proxied request traveled inside Google's network. Use the proxy-based global option for lowest handshake latency and multi-region distribution, and use passthrough when original packets, source IP, or non-HTTP protocols must be preserved.
Exam tip: Ping to a global external Application Load Balancer measures only to the nearby GFE, about 1 ms in the Germany test, while curl TTFB measures true latency to the backend, about 123 ms versus 230 ms without edge termination.
Cloud CDN and the Global Front End work together so cacheable content is served from edge PoPs instead of the backend region. When a user request arrives, traffic flows from the user to the closest GFE, then through Google Cloud Armor edge security policies, then to the Cloud CDN cache. If the request is allowed and is a cache hit, content returns directly from the edge; if it is a miss, backend security policies evaluate the request and forward it to the backend server or deny it.
This ordered flow reduces visible latency because many round trips end at the edge:
Cloud CDN supports HTTP/2 and QUIC plus routing and caching controls, which helps applications scale globally while reducing bandwidth and frontend compute costs. Choose this combination for a single-region backend serving global users, because hits avoid the long edge-to-region trip entirely.
Components of end-to-end latency are the four segments between the user and the application, and region choice can only shorten the segments inside Google's control. Fiber sets a physical floor because light travels about 200,000 km per second in fiber, so 7,500 km from Frankfurt to us-central1 needs 75 ms round-trip ideally and about 112.5 ms in a near-ideal real deployment at 1.5 times ideal. Ping checks reachability but does not show application latency for proxied load balancers, while curl TTFB shows delay before the first byte of the HTTP response arrives.
The four latency segments are:
Because last mile and app processing are often large, region optimization matters most when those segments are small, such as real-time games hurt by a few milliseconds, and matters less for high-threshold mobile or asynchronous apps that tolerate 100 ms or more.
Global deployment patterns for latency trade lower user latency against the complexity of replicating data across regions. A single-region deployment keeps frontend and backend in one region and is often best even for global users, because Cloud CDN and global load balancing cut latency without multi-region complexity. A distributed frontend with a single-region backend places frontend servers near users but keeps the database central, which helps only when requests need zero or few central round trips, such as with intelligent frontend caching or asynchronous writes.
A fully distributed frontend and backend in multiple regions answers every request locally for minimum latency, but all data must be stored locally through full replication. Spanner supports this model with multi-regional configurations, including a three-continent option with read-write replicas in the United States and read replicas in Europe and Asia for low-latency reads. Multiple parallel apps offer a middle path by directing users to separate regional frontends and backends by country domain, game room, or account location, sharing only a small fraction of data such as product catalogs or lobby services.
High-Performance Compute and Storage Configuration is the practice of matching machine type, disk size, and disk type to required input/output operations per second (IOPS), throughput, and memory. In Google Cloud, storage capacity sets the default IOPS and throughput for an instance, while machine type sets the maximum the instance can reach. You can modify these values with custom IOPS and throughput when using Hyperdisk Balanced.
How machine type and disk size control performance comes down to defaults versus limits. Storage capacity determines the starting default performance, and a larger disk automatically receives higher default IOPS and throughput up to a cap. The machine type, based on its vCPUs, determines the absolute maximum IOPS and throughput, so an instance can never exceed its machine-type limit even with a very large disk. For Cloud SQL (opens in a new tab) instances using Hyperdisk Balanced, the minimum is 3,000 IOPS and 140 MiB/s regardless of machine type.
Exam tip: Disk size sets the default IOPS and throughput, while machine type sets the maximum IOPS and throughput.
Maximum limits by machine series grow with vCPU count, but high-availability storage has lower throughput ceilings at large sizes. The C4A machine series scales from 50,000 IOPS and 800 MiB/s at 2 vCPUs to 160,000 IOPS and 2,400 MiB/s at 48 to 72 vCPUs. The C4 machine series for the sizes shown reaches 50,000 IOPS with 400 MiB/s at 2 to 4 vCPUs and 800 MiB/s at 8 vCPUs. The N4 machine series starts at 15,000 IOPS with 240 to 480 MiB/s at 0 to 15 vCPUs, then scales to 80,000 to 160,000 IOPS and 1,200 to 2,400 MiB/s at 16 vCPUs and above.
The three series compare as follows for Hyperdisk Balanced performance:
| Machine series | Maximum IOPS | Maximum throughput |
|---|---|---|
| C4A machine series | 160,000 IOPS at 48+ vCPUs | 2,400 MiB/s, or 1,200 MiB/s with high availability (HA) |
| C4 machine series | 50,000 IOPS at 2 to 8 vCPUs | 400 MiB/s at 2 to 4 vCPUs, 800 MiB/s at 8 vCPUs |
| N4 machine series | 160,000 IOPS at 48+ vCPUs | 2,400 MiB/s, or 1,200 MiB/s with HA |
High availability reduces the ceiling across series. For example, a 32-vCPU C4A instance allows 120,000 IOPS and 2,400 MiB/s with Hyperdisk Balanced but only 100,000 IOPS and 1,200 MiB/s with Hyperdisk Balanced High Availability. The same pattern appears on N4, where 48 or more vCPUs allow 160,000 IOPS and 2,400 MiB/s, but only 100,000 IOPS and 1,200 MiB/s with HA. Choose a larger vCPU count when benchmarks need higher IOPS or throughput, and plan for lower HA throughput when zonal protection is required.
Default performance by disk size grows with provisioned capacity until it hits the machine-type maximum. A 20 to 256 GB disk starts at 4,000 IOPS and 170 MiB/s, a disk over 256 to 512 GB starts at 7,000 IOPS and 240 MiB/s, and a disk over 512 GB to 1 TB starts at 10,000 IOPS and 500 MiB/s. A disk over 1 TB to 4 TB starts at 16,000 IOPS and 750 MiB/s. Very large disks are capped by the machine type, with disks over 4 TB to 32 TB defaulting to the minimum of 70,000 IOPS or the machine-type maximum and the minimum of 1,200 MiB/s or the machine-type maximum, while disks over 32 TB to 64 TB default to the minimum of 100,000 IOPS or the machine-type maximum with the same throughput rule.
Choosing a Hyperdisk type for the workload means matching durability, replication, and IOPS shape to the application. Google Cloud Hyperdisk (opens in a new tab) is durable network-attached block storage for Compute Engine virtual machine (VM) instances, containers, and bare metal instances. Data flows from the instance to the Hyperdisk volume over the network, and the volume behaves like a physical disk attached to the instance. Hyperdisk offers higher IOPS and throughput than Persistent Disk by using Titanium storage offload technology, and performance can be increased or decreased while the volume is in use.
Available Hyperdisk types are designed for different workload shapes:
Hyperdisk volumes can be detached from one instance and attached to another, shared by multiple VMs with simultaneous read-write or read-only access for high-availability workloads, and pooled in bulk through Hyperdisk Storage Pools to avoid overprovisioning. To survive a regional outage, Asynchronous Replication continuously copies data from Hyperdisk Balanced, Hyperdisk Balanced High Availability, and Hyperdisk Extreme volumes in one region to a replica in a secondary region, where the data can be failed over. Built-in torn-write protection prevents corruption if a write is interrupted by power failure or crash.
Understanding throughput, latency, and bottlenecks starts from their inverse relationship for database systems such as PostgreSQL. Throughput stays flat or rises with load until a bottleneck appears, then operations take longer and operations per second fall. The latency curve stays relatively constant until resource contention creates friction, and the inflection point where latency rises generally matches the point where throughput flattens.
Typical reasons throughput levels off or drops include:
A latency histogram groups request latencies into buckets to reveal the long tail and multiple operating modes. Most requests may complete in under 100 milliseconds, with a small tail of longer requests caused by storage access, lock contention, or network differences. A bimodal histogram shows two clusters, for example most requests under 100 milliseconds and a second cluster at 401 to 500 milliseconds, which can reflect buffer-pool hits versus storage reads, different client network latencies, time-of-day application logic, sporadic lock contention, or spikes in client activity.
Lowering end-to-end latency with placement and networking means controlling the segments that region and tier choices can change. Last-mile latency from the user to the internet service provider (ISP) is 1 to 10 ms on modern networks, 10 to 60 ms on DSL and cable, and 100 to 500 ms on 3G cellular. Traffic then travels from the user to the Google edge point of presence (PoP), where global load balancing terminates TCP and SSL sessions and Cloud CDN serves cached results, then from the edge PoP to the Compute Engine region over Google's network, plus the app's own processing time.
Google Cloud Network Service Tiers control how internet traffic enters and leaves Google's network. Premium Tier enters the high-performance Google network at the PoP closest to the sender, travels within Google's network to the Compute Engine VM, and exits at the PoP closest to the destination, reducing hops and latency. Standard Tier delivers traffic over transit ISPs from Google Cloud regions with higher latency. For globally distributed users with backends in more than one region, choose Premium Tier, and place tightly coupled tiers such as SAP application servers and databases in the same region to decrease latency.
vllm:gpu_cache_usage_perc verifies traffic ingestion.LeaderWorkerSet/vllm target when num_requests_waiting or gpu_cache_usage_perc crosses its averageValue threshold, triggering GKE scheduling and node pool expansion.Use num_requests_waiting when optimizing for throughput and cost and latency targets are met at maximum throughput, and use gpu_cache_usage_perc for latency-sensitive workloads where queue-based scaling is not fast enough. The HPA target uses averageValue which must be set experimentally for each model and latency goal, and gpu_cache_usage_perc measures KV cache utilization which correlates to requests processed per inference cycle.
Premium Tier enters Google’s private high-performance network at the point of presence closest to the sender and exits at the point of presence closest to the destination, while Standard Tier travels over transit internet service providers and enters Google’s network near the destination region with higher latency. Choose Premium Tier when serving global users from multiple regions or when using global load balancing because global load balancing requires Premium Tier.
It terminates Transmission Control Protocol or Transport Layer Security sessions at Google Front Ends (GFEs) at the edge and proxies HTTP(S) traffic to healthy backends inside Google’s network. In a test with backends only in us-central1 and a user in Germany, it showed about 1 ms ping to the nearby GFE and about 123 ms Time To First Byte compared to about 230 ms without edge termination because only the proxied request traveled across the ocean.
Storage capacity determines the starting default IOPS and throughput while the machine type based on vCPUs determines the absolute maximum that the instance can never exceed even with a very large disk. The C4A machine series scales to 160,000 IOPS and 2,400 MiB/s at 48 or more vCPUs, but high availability lowers the throughput ceiling to 1,200 MiB/s at large vCPU counts.
An e-commerce company runs a microservices-based checkout system deployed across Google Kubernetes Engine (GKE) and Cloud Run. During peak shopping events, end users report intermittent latency spikes that violate response time Service Level Objectives (SLOs), accompanied by sudden increases in cluster compute consumption. The operations team needs to pinpoint which specific inter-service calls introduce the delays and identify the exact application functions consuming excessive CPU and memory in production.
Which observability strategy should you implement to diagnose these performance bottlenecks?
Create an aggregated sink in Cloud Logging to export application logs to BigQuery, and build Looker Studio reports to analyze function call stacks and memory allocations.
Instrument the microservices with Cloud Trace to capture distributed request flows across services, and deploy Cloud Profiler agents to identify resource-intensive functions and call stacks in production.
Configure Cloud Monitoring custom dashboards with high-frequency metric collection intervals, and enable GKE usage metering to isolate function-level performance bottlenecks.
Enable Active Assist Recommender on the clusters and configure Horizontal Pod Autoscaling (HPA) using standard CPU utilization thresholds.
Last updated