Prepare and test your skills
Prepare and test your skills
Configuring autoscaling policies and dynamic resource allocation for containerized serving means setting replica limits and scaling signals so inference capacity grows and shrinks with throughput while meeting latency objectives. On Google Cloud, Vertex AI Endpoints (opens in a new tab), Cloud Run (opens in a new tab), and Google Kubernetes Engine (GKE) (opens in a new tab) each scale containerized model servers horizontally. They differ in how they obtain graphics processing units (GPUs) and tensor processing units (TPUs), handle cold starts, and expose custom metrics for scaling.
Autoscaling signals for inference throughput are the request, saturation, and latency metrics that trigger replica and node scaling. The four golden signals provide the base framework: latency as time to respond, traffic as request volume, error rate as failed requests, and saturation as use of central processing unit (CPU), memory, and GPU or TPU accelerators. Saturation shows how close the system is to capacity limits. The model observability dashboard exposes model queries per second (QPS), token throughput, and first-token latencies collected by Cloud Monitoring (opens in a new tab), while GKE and Cloud Run add processor usage, memory usage, network throughput and latency, and disk input/output.
Generative AI serving needs accelerator-aware and token-aware signals beyond CPU use. Request concurrency appears as QPS and requests per second, saturation appears as GPU and TPU utilization and per-process metrics from NVIDIA Data Center GPU Manager (DCGM) such as utilization, memory usage, and temperature, and custom latency thresholds appear as Time to First Token (TTFT), Time Per Output Token (TPOT), Inter-token latency (ITL), Normalized Time Per Output Token (NTPOT), and end-to-end request latency. Metrics flow from vLLM and NVIDIA Triton Inference Server pods to Google Cloud Managed Service for Prometheus and then into Cloud Monitoring, where dashboards and alerting policies compare them to objectives such as 95th percentile latency below 300 ms and TTFT below 500 ms for 99% of requests. Autoscaling behavior itself must be monitored to verify that replica counts respond correctly to traffic changes.
Exam tip: Requests per second is not reliable for large language model (LLM) throughput because it varies widely with context length; compare LLM serving capacity with output tokens per second, input tokens per second, or total tokens per second instead.
Platform scaling controls define which layer owns replica scaling versus node provisioning across Vertex AI Endpoints, Cloud Run, and GKE. Gemini Enterprise Agent Platform (Agent Platform) endpoints provide managed training and serving where Google manages infrastructure and the operator configures machine types and scaling policies, including endpoints deployed through Terraform. Cloud Run provides container instance scaling for agents and fine-tuned models, including a GPU-enabled service. GKE provides self-managed control where the operator manages clusters, node pools, and pods but gains fine-grained control over accelerators, topology, and serving frameworks.
The three platforms split dynamic allocation by operational control and accelerator need:
| Platform | Scaling unit and control | Accelerator and workload fit |
|---|---|---|
| Vertex AI Endpoints | Managed endpoint replicas with configured machine types and scaling policies | Managed inference without dedicated GKE GPU or TPU quotas when consuming models through the Vertex AI API |
| Cloud Run | Container instances that scale with request concurrency | Low-GPU agent workloads, fine-tuned models, and bursty traffic where management efficiency matters |
| Google Kubernetes Engine (GKE) | Pod autoscaling plus node autoscaling with custom node pools | GPU-intensive generative AI inference with A100, H100, H200, B200, or L4 GPUs and TPU-backed vLLM or Triton servers |
Choose Agent Platform endpoints when managed lifecycle and Model Monitoring integration matter most, choose Cloud Run when operational simplicity and rapid scale-out for lighter models matter most, and choose GKE when control over accelerator type, colocation, and model-aware routing matters most. Ray on Agent Platform (opens in a new tab) fits advanced cases by abstracting distributed training and serving across managed infrastructure.
Model-aware routing and accelerator obtainability determine whether scaled replicas actually receive suitable GPU or TPU capacity under fluctuating demand. GKE Inference Gateway deploys models behind a load balancer with model-aware routing that prioritizes instances with GPU and TPU accelerators for compute-intensive generative AI and LLM inference. Traffic flows from the gateway load balancer to the healthiest model instance selected by model metrics, so detailed health checks must assess model status and not only pod liveness. The gateway depends on metrics integration where data flows from vLLM or Triton servers through Google Cloud Managed Service for Prometheus into Cloud Monitoring.
Dynamic allocation extends beyond replica counts to node obtainability and cost controls. GKE custom compute classes define compute profiles with fallback policies for autoscaling, for example a profile that gives higher priority to reserved GPU or TPU instances and then falls back to cost-efficient Spot virtual machines (VMs) if reserved resources are temporarily unavailable. For diverse orchestration, a centralized load balancer routes requests with low GPU needs to Cloud Run and routes complex GPU-intensive tasks to GKE, with Cloud Service Mesh handling inter-service communication and policy management. Global users needing low latency receive traffic through a global external Application Load Balancer that distributes load across regions.
Resource selection balances stable capacity against preemptible and reserved capacity:
Cold-start mitigation and replica boundary tradeoffs decide how minimum replicas, maximum replicas, and scale-to-zero behavior interact with strict latency objectives and cost efficiency. A scaled-from-zero or newly scaled-out replica must load multi-gigabyte model weights before it can serve, so startup latency directly threatens objectives such as 99.9% successful responses, p95 latency below 300 ms, and TTFT below 500 ms for 99% of requests. Minimum replica boundaries keep warm capacity to absorb spikes without weight-loading delay, maximum replica boundaries cap cost and protect quotas for Compute Engine, Agent Platform, and Cloud Storage, and scale-to-zero eliminates idle cost but forces the next request to pay full initialization.
Weight-loading performance on GKE depends on the storage system that serves model files to scaling pods:
Capacity planning ties these choices together by using Cloud Monitoring to analyze historical utilization, projecting future needs, load testing to find bottlenecks, and proactively requesting quota increases with forecasting data. Rightsizing, Spot VMs for fault-tolerant batch work, and autoscaling then optimize usage based on demand, while quota-usage alerts warn when consumption approaches limits before peak events block scale-out.
Managing provisioned capacity, rate limits, and traffic routing for enterprise agent platforms means reserving deterministic throughput where it matters, staying within API quotas, and spreading inference traffic so spikes do not cause cascading failure. On Gemini Enterprise Agent Platform (opens in a new tab), this combines Standard pay-as-you-go for elastic capacity, Provisioned Throughput (PT) for reserved capacity, quota-aware autoscaling of Agent Platform endpoints, and multi-region routing through regional endpoints or the global endpoint.
Provisioned Throughput (PT) is a throughput reservation for production generative AI workloads that require consistent, predictable performance and cost. PT guarantees reserved capacity for business-critical, consistently high-throughput workloads and provides deterministic weekly or monthly cost structures for budget planning, which is critical for time-sensitive agent experiences. Choose PT when latency consistency and guaranteed token throughput matter more than paying only for what is used.
Standard pay-as-you-go is the elastic default where capacity is automatically managed across all users in a region without manual quota requests. Under this model, traditional quota usage does not appear in the Quotas & System Limits tab and may show as 0% on the API dashboard even while requests are succeeding. Monitor actual consumption in the Agent Platform Model Garden (Monitoring) dashboard instead.
| Model | Capacity guarantee | Cost shape | Choose when |
|---|---|---|---|
| Standard pay-as-you-go | Shared regional capacity, automatically managed | Pay for real-time use, no reservation | Traffic is variable and can tolerate shared-capacity variability |
| Provisioned Throughput (PT) | Reserved throughput for purchased models | Deterministic weekly or monthly commitment | Workload is business-critical, consistently high-throughput, and time-sensitive |
PT availability depends on the model. Supported Google models, throughput values, purchase increments, and burndown rates are listed on the PT supported-models page, while PT for partner models such as Anthropic Claude models cannot be ordered in the console and must be purchased by contacting Google. Provisioned Throughput cost planning (opens in a new tab) favors weekly or monthly commitments for stable production load over elastic pay-as-you-go.
Quotas and rate limits on Gemini Enterprise Agent Platform define how many requests and tokens the serving backend will accept before it returns error code 429 (Resource Exhausted). If the number of requests exceeds allocated capacity, the platform returns 429, which signals that the client has hit queries-per-minute (QPM) limits, token throughput limits, or custom-model serving quota. For example, Gemini 2.5 Pro has a 10 queries per minute (QPM) default limit under standard capacity.
For online inference deployed as a DeployedModel on an Agent Platform endpoint, throughput scaling is controlled by DedicatedResources. The service automatically adjusts replica count between dedicatedResources.minReplicaCount and dedicatedResources.maxReplicaCount based on traffic and autoscaling metrics such as CPU utilization and replica utilization. Configure a minimum of two replicas for high availability (HA), and note that minReplicaCount must be at least 1 unless Scale To Zero in Preview is used. Quota and limit details (opens in a new tab) define the default QPM and token limits that trigger 429 responses.
By default, Agent Platform scales replicas up or down so utilization matches a 60% target value. Without dedicated GPUs, the target applies to CPU usage; with machineSpec.accelerator_count greater than 0, the target applies to whichever is higher, CPU or GPU usage. Endpoints scale per machine, but custom model serving quota is charged per CPU or GPU based on real-time usage. If the sum of maxReplicaCount across deployments exceeds project quota, autoscaling can stall, so a deployment on a2-highgpu-2g counts as 24 CPUs and 2 GPUs per active replica against quota.
Deployment success also depends on replica availability. The deployment operation is marked successful only when inference nodes reach minReplicaCount before the request timeout, otherwise it is marked failed and resources are released. Setting dedicatedResources.requiredReplicaCount lower than minReplicaCount allows a partially successful deployment: the operation returns successful once requiredReplicaCount is reached, deployment continues toward minReplicaCount, and any replicas that never become ready are reported in DeployedModel.status.message.
Exam tip: Error 429 means requests exceeded allocated standard capacity — reserve throughput with Provisioned Throughput for consistently high-throughput workloads rather than repeatedly retrying into the same limit.
Multi-region traffic routing for enterprise agent serving distributes inference deployments across locations and routes requests to healthy capacity. For high availability and low latency in real-time inference and generative AI, distribute deployments across multiple locations by deploying models to multiple Agent Platform endpoints across Google Cloud regions or by using the global endpoint. Traffic then flows from clients to the global load-balancing layer, which routes each request to an available regional endpoint.
This topology isolates zonal and regional failures. If data can be shared across regions or countries, use the global endpoint for model availability and resilience; otherwise keep separate regional endpoints to enforce residency. Run multiple instances of the agent application behind a load balancer so requests are distributed across instances, which reduces latency and timeouts and preserves availability when one instance or zone fails. For containerized agents, Cloud Run synchronously stores data across zones and load-balances traffic across zones within a region, while Google Kubernetes Engine (GKE) regional clusters spread nodes across multiple zones. Agentic architecture choices (opens in a new tab) guide when to use regional endpoints, the global endpoint, Cloud Run, or GKE for agent serving.
Resilience also depends on redundant state. The serving backend depends on highly available dependencies, including:
Decouple agent state from runtime by storing context in an external datastore such as Memory Bank, Memorystore for Redis, or Cloud SQL, so an agent can restart and recover context after an outage.
Proactive capacity planning for enterprise agent platforms forecasts GPU, TPU, quota, and token-throughput needs before spikes arrive. Plan capacity from historical telemetry in Cloud Monitoring (opens in a new tab) and Cloud Logging, including GPU or TPU utilization and throughput rates, then analyze trends in BigQuery or Looker Studio to predict when and where accelerators and quota increases are needed. Increase resource quotas before production peaks rather than during them.
Validate estimates with rigorous load testing in a production-like environment:
Maintain continuous observability after deployment. Use the model observability dashboard to view Cloud Monitoring metrics such as model queries per second (QPS), token throughput, and first-token latency, and set proactive alerts for agent quality and query volume. This closed loop — forecast from telemetry, validate under stress, reserve PT and quota, then monitor QPS and token throughput — keeps high-throughput Gemini Enterprise and agentic pipelines resilient during extreme spikes. Client-side retries with exponential backoff and traffic shedding prevent a spike from cascading when limits are reached.
Optimize Throughput and Concurrency in High-Performance Inference Frameworks is the practice of tuning containerized model servers and their scaling backend to maximize tokens per second or queries per second (QPS) without breaching latency objectives or accelerator memory limits. On Google Cloud, Google Kubernetes Engine (GKE) (opens in a new tab) hosts specialized runtimes such as vLLM, NVIDIA Triton Inference Server (Triton), and TorchServe, while GKE Inference Gateway and Google Cloud Managed Service for Prometheus connect runtime metrics to model-aware routing and autoscaling decisions.
Inference throughput and latency metrics define the tradeoff between concurrency and responsiveness for large language model (LLM) serving on GKE. Latency metrics isolate the prefill phase from the decode phase, while throughput metrics count how much work the server completes per second under concurrent load. Throughput rises with concurrency only to the inflection or saturation point, where throughput stops increasing and latency starts increasing dramatically.
The benchmarking metrics for inference performance on GKE compare request counts against token counts:
| Throughput metric | What it measures | When to use it |
|---|---|---|
| Requests per second | Total requests served per second | General online load, but not reliable for LLMs because it varies widely with context length |
| Output tokens per second | total_output_tokens_generated_by_server / elapsed_time_in_seconds | Decode-bound generation throughput |
| Input tokens per second | total_input_tokens_generated_by_server / elapsed_time_in_seconds | Prefill-heavy workloads |
| Tokens per second | total_tokens_generated_by_server / elapsed_time_in_seconds, counting input plus output tokens | Comparison of workloads with high prefill versus high decode times |
Latency metrics pair with throughput to guard concurrency tuning. Time to First Token (TTFT) measures milliseconds to generate the first token, Time Per Output Token (TPOT) measures (request_latency - time_to_first_token) / (total_output_tokens - 1), Normalized Time Per Output Token (NTPOT) measures request_latency / total_output_tokens, Inter-token latency (ITL) measures time between each individual token generation aggregated as mean, median, and p90, and request latency measures end-to-end milliseconds to complete a request.
Exam tip: Requests per second is not reliable for LLM throughput because context length changes the cost per request; use output tokens per second or total tokens per second to compare prefill-heavy versus decode-heavy concurrency settings.
Containerized high-performance runtimes are the model servers that execute inference inside Pods on GPU or TPU nodes and expose the concurrency controls and metrics used for scaling. GKE supports vLLM, Triton, TorchServe, Hugging Face Text Generation Inference (TGI), Ray Serve, and MaxDiffusion across GPU and TPU tutorials, with vLLM as the primary LLM server for single-host GPU, multi-host GPU, and TPU Trillium deployments. A vLLM engine can dynamically use TPUs and GPUs to address fluctuating capacity and performance needs through vLLM GPU/TPU fungibility. Open-model serving on GKE (opens in a new tab) provides pre-configured architectures for deploying these runtimes with Terraform.
Each runtime depends on accelerators, fast weight access, and metric export to sustain high QPS. Model weights flow from storage to each Pod at startup, so slow loading delays scale-out and reduces effective concurrency. Cloud Storage FUSE mounts model weights for deployments with limited node scaling, Google Cloud Managed Lustre (Managed Lustre) provides high-throughput and low-latency file access from multiple Pods simultaneously and can attach to 10,000 or more Pods, and Google Cloud Hyperdisk ML (Hyperdisk ML) offers dedicated block storage that can attach to up to 2,500 Pods for massive-scale, consistent low-latency access. Runtime metrics then flow from the serving framework to Cloud Monitoring (opens in a new tab) through Google Cloud Managed Service for Prometheus, which carries vLLM and Triton LLM metrics for dashboards, alerts, and gateway routing. Inference request handling (opens in a new tab) explains how online predictions move from client request to model server response.
Model-aware scaling and traffic distribution directs concurrent requests to the backend instances that can serve them with the lowest latency and highest accelerator utilization. GKE Inference Gateway deploys models behind a load balancer with model-aware routing that prioritizes instances with GPU and TPU accelerators for compute-intensive generative AI and LLM inference. Traffic flows from clients to the gateway, the gateway selects a healthy model instance based on model metrics and detailed health checks that assess model status, and the selected Pod executes inference on its accelerator.
The gateway composes with broader GKE scaling controls to handle fluctuating throughput. GKE custom compute classes define compute profiles with fallback policies for autoscaling, such as higher priority for reserved GPU or TPU instances with fallback to cost-efficient Spot VMs when reserved resources are unavailable. A centralized load balancer can route low-GPU requests to Cloud Run and GPU-intensive tasks to GKE, Cloud Service Mesh manages inter-service communication and policy, and a global external Application Load Balancer distributes traffic from global users who need low latency.
Selecting optimized configurations at saturation point means choosing the tested combination of model, model server, model server version, and accelerator that delivers the target TTFT and tokens per second before latency climbs. Inference Quickstart (opens in a new tab) generates optimized profiles and reports performance observed at the inflection point, including output tokens per second, NTPOT, TTFT, and cost per million input and output tokens in USD. Pricing defaults to the us-east5 region, except A3 machine configurations that default to us-central1.
Profile selection follows an ordered evaluation workflow:
gcloud container ai profiles models list to view supported families such as Gemma, Llama, Mistral, DeepSeek, and Qwen.gcloud container ai profiles list with filters such as --model, --pricing-model, and --target-ttft-milliseconds to compare instance type, accelerator, model server, and saturation metrics.gcloud container ai profiles benchmarks list with --model and --model-server to view performance metrics from benchmarks run at different request rates.Choose the profile whose saturation throughput meets QPS goals while TTFT and NTPOT remain within service level objectives (SLOs), such as 95th percentile inference latency below 300 ms and TTFT below 500 ms for 99% of requests. Dynamic batching, tensor parallelism, key-value (KV) cache sizing, worker counts, and asynchronous input/output handling are then tuned so concurrency increases without exceeding accelerator memory.
Observability for concurrency control links runtime saturation signals to scaling actions for vLLM, Triton, and TorchServe backends. The model observability dashboard views metrics collected by Cloud Monitoring such as model QPS, token throughput, and first token latencies. Those application signals combine with infrastructure metrics for CPU, GPU, and TPU usage, memory usage, network throughput and latency, and disk input/output from GKE nodes and managed serving environments.
Monitoring applies the four golden signals of latency, traffic, error rate, and saturation, where saturation shows utilization of CPU, memory, and GPU or TPU accelerators and indicates how close the system is to capacity limits. Metrics flow from runtimes and Gemini Enterprise Agent Platform (Agent Platform) endpoints into Cloud Monitoring dashboards and alerts, logs flow into Cloud Logging for troubleshooting, request flows across distributed inference pipelines flow through Cloud Trace, code-level bottlenecks are identified with Cloud Profiler, and per-process GPU utilization, memory, and temperature are gathered with NVIDIA Data Center GPU Manager (DCGM). SLOs such as 99.9% successful API responses and harmful-output rate below 0.1% then define when concurrency is too high and additional replicas or different accelerator profiles are required.
Use Provisioned Throughput (PT) for business-critical, consistently high-throughput and time-sensitive workloads that require reserved capacity with deterministic weekly or monthly cost structures. Use Standard pay-as-you-go for variable traffic that can tolerate shared regional capacity variability, where capacity is automatically managed across all users in a region without manual quota requests and consumption is monitored in the Agent Platform Model Garden Monitoring dashboard.
The service automatically adjusts replica count between dedicatedResources.minReplicaCount and dedicatedResources.maxReplicaCount based on traffic so utilization matches a 60 percent target, which applies to CPU usage or to whichever is higher, CPU or GPU usage, when machineSpec.accelerator_count is greater than 0. Configure a minimum of two replicas for high availability, and if requests exceed allocated queries-per-minute or token limits the platform returns error code 429 Resource Exhausted.
Choose Cloud Storage FUSE for deployments with limited node scaling because it mounts model weights from Cloud Storage with caching and parallel downloads. Choose Google Cloud Managed Lustre for inference demanding consistent scale-out with high-throughput, low-latency access from multiple pods simultaneously because it can attach to 10,000 or more pods, and choose Google Cloud Hyperdisk ML for massive-scale scenarios demanding consistent low-latency access because it can attach to up to 2,500 pods.
Requests per second is unreliable for large language model throughput because cost per request varies widely with context length. Compare LLM serving capacity with output tokens per second for decode-bound generation, input tokens per second for prefill-heavy workloads, or total tokens per second counting input plus output tokens, guarded by Time to First Token (TTFT), Time Per Output Token (TPOT), Normalized Time Per Output Token (NTPOT), Inter-token latency (ITL), and end-to-end request latency.
Last updated