Tuning ML Models for Training and Serving

Optimize Distributed Training and Hyperparameter Search Using Vertex AI is the practice of using automated black-box search and parallel training to find the best model settings with the least time and cost. Agent Platform Vizier (opens in a new tab) is the independent optimization service at the center of this work, and it suggests parameter combinations that balance predictive performance against inference latency, throughput, and infrastructure cost. Training workers run the suggested trials and report measurements back, so the optimizer stays decoupled from the compute that executes each trial.

1.1Black-Box Optimization Scope

Black-box optimization scope covers any evaluable system where the objective function is unknown or too costly to evaluate directly because of system complexity. Agent Platform Vizier can tune hyperparameters such as neural network depth, width, learning rate, and batch size for a TensorFlow recommendation model, and it can also tune non-ML parameters such as background color, font size, and link color for a subscription button. This breadth makes Agent Platform Vizier usable with training jobs or with other systems, including multi-cloud systems, because the service only suggests values and evaluates reported results.

Agent Platform Vizier use cases show the same pattern of expensive evaluation with configurable inputs:

• Optimize learning rate, batch size, and other hyperparameters of a neural network recommendation engine.
• Optimize application usability by testing different arrangements of user interface elements.
• Minimize computing resources for a job by identifying an ideal buffer size and thread count.
• Optimize ingredient amounts in a recipe to produce the most preferred version.

1.2Studies, Trials, and Measurements

Studies, trials, and measurements are the three objects that define an optimization problem and record its results. A study configuration is the definition of the problem, including the result to optimize and the hyperparameters or parameters that affect that result. A study is the implementation of a study configuration that conducts experiments called trials, while a trial is a specific set of input values that produces a measured outcome. A measurement is the measured outcome of a trial, where each measurement can contain one or more metrics and each trial can contain one or more measurements taken over time.

Agent Platform Vizier suggests input values for each trial but does not run trials for you. A study continues until it reaches a set limit of trials or you interrupt the study, while a trial continues until you indicate that it is either finished or infeasible. You can add a new measurement to a trial at any point before the trial is completed, so intermediate training metrics can flow from the training worker to the study before completion.

1.3Search Algorithms for Hyperparameter Search

Search algorithms for hyperparameter search control how Agent Platform Vizier selects the next trial values from the parameter space. The default algorithm applies Bayesian optimization for a more effective search over the parameter space. If you do not specify an algorithm, Agent Platform Vizier uses ALGORITHM_UNSPECIFIED, where Agent Platform chooses between Gaussian process bandits, linear combination search, or their variants.

1.4Vizier Service Versus Custom Training Hyperparameter Tuning

Vizier service versus custom training hyperparameter tuning is the distinction between an independent optimizer and a built-in training feature. Agent Platform Vizier is an independent service for optimizing complex models with many parameters for both ML and non-ML use cases. Hyperparameter tuning for custom training is a built-in training feature that uses Agent Platform Vizier for training jobs to determine the best hyperparameter settings for an ML model. In the independent pattern, your training job or external system runs each trial suggested by Agent Platform Vizier and then reports measurements back.

Agent Platform Neural Architecture Search is distinct from hyperparameter tuning and should not be combined with it. Agent Platform Neural Architecture Search searches neural architectures for accuracy, latency, memory, or a custom metric across a search space that can reach 10^20 choices, while hyperparameter tuning adjusts settings such as learning rate or optimizer settings. Use traditional methods such as hyperparameter tuning first, and reserve architecture search for enterprise experimentation after those gains are exhausted.

1.5Holistic Training Performance and Resource Planning

Holistic training performance and resource planning looks beyond forward and backward propagation to data input, failures, quota, and cost. Training execution for forward and backward propagation is only a fraction of the lifecycle of an ML job, so resource-efficiency metrics such as model FLOPs utilization (MFU) (opens in a new tab) alone do not explain overall performance. As model size and cluster size grow, failure rate and infrastructure cost can rise, and the cost of failure can increase quadratically. Bottlenecks can occur in data I/O at the storage layer or in model limitations even on powerful GPU accelerators, so design choices must be matched to measured bottlenecks and business requirements.

A typical training job follows an ordered cyclical flow where each interruption repeats the preceding flow:

1. Cluster creation: Worker nodes are provisioned for the job.
2. Initialization: Training is initialized on the worker nodes.
3. Training execution: Resources are used for forward or backward propagation.
4. Interruption: Training is interrupted during model checkpointing or due to worker-node preemptions.

Accelerator quota directly constrains how many parallel trials and workers can run, which controls search time and cost. By default, the Custom model serving TPU v5e cores per region quota is 4, which is sufficient for serving a Llama 3.1 8B model, while a Llama 3.1 70B model requires 16 TPU v5e cores in us-west1. The Custom model serving Nvidia A100 80GB GPUs per region quota is 0, so at least 1 GPU must be requested for an 8B model and at least 4 for a 70B model. Large searches multiply this effect because GPU hours equal total trials times training time per trial times GPUs per trial, so parallel trials reduce wall-clock time but do not reduce total GPU hours.

1.6Tuning Models for Serving in Production

Tuning models for serving in production balances cost-effectiveness, throughput, and latency because traffic patterns differ by use case. Chat applications require minimized latency for interactive experience, while recommendation applications may require maximized throughput. Prompt length for input and generated length for output directly affect both performance and cost, so short, direct prompts with sufficient context and requests for concise responses reduce work per request.

Prefix caching (opens in a new tab) reuses computations from previously generated text to eliminate redundant processing and reduce Time-To-First-Token (TTFT) for requests with common prompt prefixes. Typical prefix caching patterns are asking different questions with the same long document as context and multi-turn chat conversations. In vLLM serving, GPU memory prefix key-value (KV) caching is fast but small, while Vertex Model Garden prebuilt vLLM also supports VM host memory prefix KV caching that is larger but requires transfer from host memory to GPU memory on cache hit. For example, on an a3-high-gpu-4g VM with 4xH100 80GB serving Llama 3.1 70B, GPU cache capacity is capped below 180GB after 140GB of model weights, while host memory cache can reach up to 936GB. When host caching is enabled, processed KV state is written to both GPU cache and host cache, and the write to host memory runs asynchronously without hurting inference request performance.

Speculative decoding reduces generation Time-Per-Output-Token latency for large language models that otherwise generate tokens one-by-one with sequential autoregressive decoding. With speculative decoding, a fast drafter efficiently guesses multiple tokens and then the larger language model verifies them in parallel. Production endpoints further control latency, throughput, and cost by selecting machine type and endpoint type, using dedicated private endpoints and optimized online serving with Private Service Connect where low latency matters, sharing resources across model deployments through model co-hosting, using reservations, Dynamic Workload Scheduler, or Spot VMs for availability, splitting traffic between model versions for A/B testing, using rolling deployment to replace models, and observing logs, inference metrics, and Model Monitoring to iterate on serving performance.

Tune Online Serving Infrastructure, Autoscaling, and Request Batching on Vertex AI is the work of sizing and scaling prediction endpoints so live traffic gets fast answers without overspending. Vertex AI Prediction (opens in a new tab) for online prediction is a regional service for near real-time scoring of live applications. A client sends input instances to a cloud-hosted model deployed to an endpoint, and the service returns predictions in the response for each instance. Tuning that endpoint means selecting the right hardware, configuring autoscaling bounds, feeding features with low latency, and planning for zone and region failures while controlling cost.

2.1Select Machine Types for Online Serving

Machine type selection for a Vertex AI endpoint is the choice of hardware that will run the serving binary for each deployed model. The deployment specifies the number and type of machines, such as different central processing unit (CPU) virtual machine (VM) types or graphics processing unit (GPU) types appropriate for the model size and compute pattern. That hardware choice directly sets the baseline for inference latency, per-node throughput, and hourly infrastructure cost. Larger or accelerated machine types increase single-node capacity but raise cost, so the selection must match model requirements before autoscaling is tuned.

2.2Configure Autoscaling for Fluctuating Traffic

Automatic scaling in Vertex AI Prediction is the control that adds and removes prediction nodes between a configured minimum and maximum. In most cases automatic scaling should be turned on by setting both minimum and maximum nodes, which lets the endpoint expand during peaks and contract when request volume falls. To ensure a high availability service level agreement (SLA), automatic scaling must be set with a minimum of two nodes so capacity remains if one node fails. The minimum therefore protects availability and warm capacity, while the maximum caps cost and quota use under spikes.

2.3Serve Low-Latency Feature Inputs to the Endpoint

Feature input serving for online prediction is the path that supplies the features passed to the endpoint with each request. Clients calling the online prediction endpoint first call a feature-serving solution to fetch feature inputs, and then call the prediction endpoint with those inputs. That solution can be one of the many Database services on Google Cloud or the Vertex AI Feature Store (opens in a new tab) online serving API, which retrieves features in real time for online prediction. Vertex AI Feature Store streaming ingestion keeps those values current by making streaming event data available for online serving within a few seconds, which matters when the latest data is a priority. Input and output handling can be further customized with custom prediction routines that control request and response format to and from the model server.

2.4Reduce Network Latency with Private Endpoints

Vertex AI private endpoints are peer-to-peer prediction endpoints that keep traffic inside the same local network. Private endpoints are particularly useful when the application making prediction requests and the serving binary are within the same local network. The connection avoids the overhead of internet routing by using Virtual Private Cloud (VPC) for direct client-to-model-server communication. Traffic therefore flows from the client application through the VPC to the model server, rather than from the client over the public internet to a public endpoint.

2.5Split Traffic and Share Resources Across Deployments

Traffic splitting on Vertex AI Prediction is the pattern of serving multiple models from the same endpoint to shift load gradually. Multiple models can be deployed to the same endpoint, for example to gradually replace an older model with a newer version while both serve traffic. Alternatively, models can be deployed to multiple endpoints, for example to separate testing and production, by sharing resources across deployments through model co-hosting. The choice controls the blast radius and resource sharing of the rollout:

Co-hosting lets separate endpoints share underlying resources, so isolation does not always require fully duplicated infrastructure.

2.6Design for Zonal and Regional Resilience

Resilience for Vertex AI Online prediction is defined by a regional deployment that load-balances across zones without replicating data across regions. Customers choose the region but not the specific zones, and the prediction service automatically load-balances workload across different zones within the selected region. Online prediction does not store customer content, so a zonal outage causes failure of the in-flight prediction request execution. A public endpoint will retry automatically while a private endpoint will not, so applications using private endpoints must incorporate retry logic with exponential backoff in client code.

When a regional failure occurs, online prediction is unavailable in that region and becomes available again only when the outage resolves, because execution is scoped to the requested region and never routes to a different region. Customers should therefore run models in multiple regions and direct traffic to a different available region during a regional outage.

2.7Balance Throughput, Latency, and Operational Cost

Cost and availability optimization for serving is the practice of matching compute commitments and proactive monitoring to workload stability. For stable 24x7 inference with fixed or predictable capacity, committed use discounts (CUDs) for VMs and accelerators lower cost for reserved capacity. For fault-tolerant evaluation and experimentation, Spot VMs reduce cost with the tradeoff that they can be preempted, while Dynamic Workload Scheduler (DWS) improves obtainability of high-demand accelerators through Flex-Start mode for complex batch work up to seven days and Calendar mode for longer reservations of specific GPUs and TPUs. Continuous telemetry from Cloud Monitoring and Cloud Logging, analyzed for GPU or TPU utilization, queries per second, token throughput, and first-token latency, supports forecasting, quota increases, and rigorous load testing with tools such as Apache JMeter to identify the concurrency level where accelerator availability limits performance.

Apply Model Compression and Hardware-Aware Compilation for Low-Latency Inference is the practice of shrinking memory footprint and compiling execution for a specific accelerator so latency drops without losing target accuracy. On Google Cloud, Google Kubernetes Engine (GKE) (opens in a new tab) divides this work between profile selection, framework runtimes such as TensorRT-LLM, vLLM, and SGLang, server batch tuning, and gateway routing that reuses computation. The tradeoff is evaluated at the inflection point where throughput stops increasing and latency starts increasing dramatically, balancing Time-to-First-Token (TTFT), Time-per-Output-Token (TPOT), output tokens per second, and cost per million tokens.

3.1Optimized Inference Profiles for Hardware-Aware Selection

Optimized inference profiles are combinations of model, model server, model server version, and accelerator that expose measured throughput, latency, and cost for a specific hardware target. The profile workflow runs from the Google Cloud CLI first to gcloud container ai profiles models list to select from families including Gemma, Llama, Mistral, DeepSeek, and Qwen, then to gcloud container ai profiles list filtered by --model, --pricing-model, and --target-ttft-milliseconds, then to gcloud container ai profiles benchmarks list for request-rate curves in CSV format. The list output reports Instance Type, Accelerator, Cost per million input and output tokens, Output Tokens per second, NTPOT in milliseconds, TTFT in milliseconds, Model Server, and Model Server Version at saturation, so an expert can compare for example a3-highgpu-1g with nvidia-h100-80gb serving openai/gpt-oss-20b with vLLM at 297 ms TTFT. All pricing is in USD and defaults to the us-east5 region, except A3 machine configurations which default to us-central1, and benchmark recipes are only supported for virtual machines (VMs) that use future reservations and were tested on clusters created with Cluster Toolkit.

3.2Framework and Accelerator Compilation Choices

Framework and accelerator choice determines how model execution is compiled and scheduled on specialized serving hardware. Reproducible inference recipes pair A3 Mega and A3 Ultra accelerators with large models such as DeepSeek R1 671B, GPT3-175B, Llama3 70B, Llama3.1 70B, Llama-3.1-405B, and Mixtral-8-7B, so the same model can be evaluated as a different compiled system on different hardware.

The available frameworks divide by role in training versus serving:

• NeMo provides pre-training recipes for models such as Llama3.1 70B and GPT3-175B on GKE.
• MaxText provides scalable training and serving baselines for large transformer models.
• SGLang provides inference serving recipes for models such as DeepSeek R1 671B on A3 Mega and A3 Ultra.
• TensorRT-LLM provides hardware-aware compiled inference for NVIDIA GPUs, maximizing execution efficiency for low-latency serving runtimes.
• vLLM provides high-throughput inference serving with engine metrics such as vllm:num_requests_waiting and batch controls such as max_num_seqs.

Choose TensorRT-LLM when serving latency on NVIDIA GPUs is the binding constraint, choose vLLM or SGLang when throughput, continuous batching, and gateway integration matter, and validate the choice against the profile inflection point rather than peak theoretical throughput.

3.3Serving Runtime Tuning and Request Routing for Latency

Serving runtime tuning is the adjustment of batch and routing behavior so the accelerator stays saturated without queueing delay that raises TTFT and TPOT. The inference server depends on batch settings including max_num_seqs and max_num_batched_tokens, which must be tuned for the specific accelerator and use-case token shape because input-to-output ratio changes the serving stack behavior. The following use cases show why one configuration cannot fit all traffic, since long inputs stress prompt processing while long outputs stress token generation.

GKE Inference Gateway (opens in a new tab) extends server tuning with request-level optimizations that reduce redundant work and protect latency-sensitive traffic. Cache Aware Routing sends requests with shared context identified from the request body to the same model replica to maximize cache hits, which reduces redundant computation and improves TTFT for conversational AI, Retrieval-Augmented Generation (RAG), and template-based workloads. Dynamic low-rank adaptation (LoRA) serving multiplexes multiple LoRA fine-tuned adapters on a common base model and common accelerator, which reduces the number of graphics processing units (GPUs) and tensor processing units (TPUs) required by densely packing models. Predicted latency-based routing uses an XGBoost model trained continuously on live traffic to route each request for TTFT and TPOT objectives, which is more accurate than static heuristics under high-variance workloads, while serving Priority drops requests with Priority less than 0 first with a 429 error when the system is under pressure and defaults to 0 otherwise.

3.4Async Buffering and Independent Autoscaling for Throughput

Async buffering is a durable queue between ingestion and inference that protects inference servers from traffic spikes while keeping accelerators fully utilized. Pub/Sub acts as the persistent first-in, first-out (FIFO) buffer with 7 to 31 days retention, the subscriber pulls only when it has capacity, and failed messages flow to a dead-letter topic after exponential backoff retries while results flow from the inference server to storage in AlloyDB or Cloud Storage. This separation lets central processing unit (CPU)-bound subscribers scale differently from GPU- or TPU-bound inference servers, so backlog does not shift contention to the model server without increasing throughput.

The request path moves through an ordered sequence where each stage depends on the previous stage having capacity:

1. Pub/Sub topic buffers: Client messages land in the Pub/Sub topic where the topic holds requests until a consumer can process them.
2. Subscriber pulls and aggregates: The subscriber pulls batches with client-side flow control and bundles multiple Pub/Sub messages into one inference request that fills the server batch size.
3. Inference Gateway exposes: The subscriber sends the aggregated batch to the Inference Gateway, which routes the batch to inference workloads.
4. Inference Server executes: The inference server, such as vLLM, processes the saturated forward pass and the subscriber acknowledges the Pub/Sub messages only after processing.
5. Storage and dead-letter handling: Results persist to AlloyDB or Cloud Storage, while messages that exhaust retries move from the subscriber to the dead-letter topic.

To keep every forward pass saturated, configure the subscriber max_messages pull setting to a multiple of the server max_num_seqs and never send one Pub/Sub message as one inference request. Scale subscribers on backlog with desiredReplicas = num_undelivered_messages / (target_latency_seconds x throughput_per_replica) using the num_undelivered_messages metric and cap growth with maxReplicas to respect GPU and TPU quota, then scale inference servers separately on engine backlog such as vllm:num_requests_waiting rather than only on CPU or memory.

Key Facts

• Agent Platform Vizier suggests trial values but does not run trials, and a study ends at a trial limit while a trial ends as finished or infeasible.
• GRID_SEARCH requires all parameters to be INTEGER, CATEGORICAL, or DISCRETE and avoids duplicate suggestions when trials exceed feasible points.
• Vertex AI Prediction automatic scaling should use a minimum of two nodes for high availability, and a public endpoint retries after a zonal outage while a private endpoint requires client-side retry with exponential backoff.
• On an a3-high-gpu-4g VM serving Llama 3.1 70B, GPU prefix KV cache is capped below 180GB after 140GB of model weights while host memory cache can reach 936GB.
• GKE Inference Gateway Cache Aware Routing sends requests with shared context to the same replica to improve Time-to-First-Token, while Pub/Sub buffering with 7 to 31 days retention lets CPU-bound subscribers scale separately from GPU-bound inference servers.