Online serving infrastructure on Vertex AI hosts trained machine learning models on managed compute resources to deliver low-latency predictions to client applications. Configuring online prediction endpoints requires balancing hardware resources, autoscaling policies, dynamic request batching, and network routing to sustain high throughput while controlling operational expenses.
Machine type selection determines the underlying compute, memory, and accelerator resources assigned to each replica in a Vertex AI Prediction deployment. Workloads can run on general-purpose central processing unit (CPU) instances such as E2 virtual machines (VMs), accelerator-optimized instances such as A2 VMs configured with NVIDIA A100 80GB graphics processing units (GPUs), or Cloud TPU v5e hardware. Selecting a machine type depends on the computational footprint of the model, memory requirements, and latency targets. The chosen machine profile defines the hardware boundary for every deployed replica serving traffic.
Autoscaling dynamically adjusts the replica count of a deployed model between a configured minimum and maximum boundary to handle shifting traffic loads. A model deployment sets minReplicaCount and maxReplicaCount parameters to define this range. Meeting the high availability service level agreement (SLA) requires setting minReplicaCount to a minimum of two nodes. Vertex AI calculates compute quota based on real-time resource consumption rather than the configured maxReplicaCount, which prevents over-reserving quota but limits automatic scale-out if the active project exhausts its available compute quota.
Server-side dynamic request batching and traffic-splitting control how inference requests flow into model replicas. Dynamic request batching groups multiple incoming client requests on the server side into a single inference pass, maximizing hardware accelerator utilization and boosting overall request throughput. Traffic splitting routes incoming requests across multiple model versions deployed to the same prediction endpoint by assigning percentage weights to each version. This mechanism enables controlled A/B testing, gradual canary rollouts, and direct performance comparisons between model iterations in production.
Network path and feature store configurations determine data ingestion latency between client callers and model prediction endpoints. Deploying prediction endpoints as Vertex AI private endpoints restricts inference traffic to a Virtual Private Cloud (VPC) network, eliminating internet routing overhead and securing communication boundaries. For feature inputs, clients can retrieve precomputed values through the Vertex AI Feature Store online serving API or ingest low-latency streaming updates via Vertex AI Feature Store streaming ingestion, which makes fresh feature values available for online predictions within seconds.
Optimizing machine learning workloads across training and serving requires systematically tuning hyperparameters, designing network architectures, and coordinating distributed hardware clusters. Vertex AI offers automated tools to evaluate multi-dimensional parameter spaces and execute distributed compute across multi-node worker pools.
Vertex AI Vizier is a managed black-box optimization service that identifies optimal parameter configurations for complex machine learning models. A user defines a study configuration that specifies target objective metrics and tunable parameters such as learning rate, batch size, and network depth. Vertex AI Vizier generates parameter combinations called trials, and the training application runs the trial and reports evaluation metrics back to the service. The service uses Bayesian optimization by default to build a probabilistic model of the search space, but it can also execute grid search across INTEGER, CATEGORICAL, or DISCRETE parameter types, or use random search to sample configurations across wide parameter spaces.
Vertex AI Neural Architecture Search (NAS) automates the structural design of neural network layers to balance model accuracy, inference latency, and memory footprint. The search operates across a two-stage lifecycle:
Neural architecture search works best when specialized engineering teams define explicit search spaces; it is not recommended for limited or highly imbalanced datasets where heavy data augmentation masks performance differences between architectures.
Serving optimizations for large language models (LLMs) improve token generation speed and overall server throughput. Prefix caching stores intermediate key-value states from previously processed prompts, eliminating redundant computation for multi-turn chats or shared system instructions and reducing Time-To-First-Token (TTFT). Serving frameworks can place prefix caches in GPU memory for faster access or in host VM memory for higher storage capacity. Speculative decoding reduces Time-Per-Output-Token latency by using a lightweight draft model to generate candidate tokens in parallel, which the primary model then validates in a single execution step.
Distributed training coordinates computation across clusters of GPUs or Tensor Processing Units (TPUs) to accelerate model convergence. To prevent input/output bottlenecks, training datasets must reside in a Cloud Storage bucket located within the same geographical region as the compute cluster. Training jobs employ mixed-precision computation—using 16-bit floating point (float16) or bfloat16 formats—to halve memory consumption and maximize hardware execution speed. Synchronizing updates across worker pools requires tuning gradient accumulation and batch sizes so network communication between nodes does not stall compute cores.
Selecting an optimization technique depends on whether the bottleneck occurs in model architecture design, training convergence, or production inference:
Low-latency inference relies on transforming model computation graphs through compression techniques and hardware-aware compilation runtimes. Deploying these optimized models across managed platforms like Google Kubernetes Engine (GKE), Cloud Run, and Vertex AI Prediction aligns model memory footprints with physical accelerator architectures.
Framework-level compilation engines and compression techniques restructure neural networks to execute efficiently on specialized hardware accelerators. Serving engines such as NVIDIA TensorRT-LLM, vLLM, SGLang, and MaxText optimize execution graphs by fusing computational kernels and managing memory layouts. Complementary compression techniques reduce model size before serving:
Hardware-aware deployments pair model architectures with specific machine profiles and high-speed network fabrics to eliminate data transfer bottlenecks. The A4 machine series features NVIDIA B200 GPUs with 180 GB of memory per accelerator, interconnected by bidirectional NVLink interfaces delivering 1,800 GBps per GPU and 14.4 TBps of aggregate system bandwidth. Distributed inference across multi-host A4 clusters communicates over RDMA over Converged Ethernet (RoCE) through NVIDIA ConnectX-7 network interface cards (NICs). Alternative hardware options include the Arm-based A4X machine series equipped with NVIDIA GB200 Grace Blackwell Superchips, as well as G2 and G4 machine series designed for single-host and cost-sensitive inference workloads.
GKE Inference Gateway routes incoming inference traffic across model server replicas based on real-time server utilization metrics and request priorities. When a client sends an inference request, the gateway extracts the model identifier using body-based routing extensions, filters the payload through Model Armor for security checks, and evaluates target replicas in an InferencePool. The gateway endpoint picker scores each replica based on key-value cache (KV-cache) utilization, queue depth, prefix match locality, and loaded Low-Rank Adaptation (LoRA) adapters. Under heavy load, an InferenceObjective resource enforces priority shedding, dropping requests with priority values below 0 using HTTP 429 status codes to preserve capacity for high-priority traffic.
Cloud Run and Vertex AI Prediction provide managed runtime environments that automatically scale containerized model instances based on live incoming traffic. Cloud Run hosts inference code in standard containers, supports streaming responses over HTTP chunked transfer, WebSockets, or HTTP/2, and can attach NVIDIA GPUs while retaining scale-to-zero capability during idle periods. When loading models larger than 90 GiB into container instances, setting the HF_HOME environment variable to /dev/shm/hf_cache directs caching into node shared memory (RAM), which prevents container crashes caused by exhausted local boot disk space. Vertex AI Prediction complements this by providing prebuilt serving containers optimized for TensorFlow, PyTorch, scikit-learn, and XGBoost on accelerator-backed hardware.
StudyCerts
StudyCertsminReplicaCount configuration parameter to a minimum of two nodes.429 status codes under system load.HF_HOME environment variable to /dev/shm/hf_cache on container hosts directs model weights into shared RAM, preventing container evictions caused by boot disk exhaustion.Vertex AI Vizier tunes external hyperparameters (such as learning rate and batch size) for a predefined, static model architecture. Vertex AI Neural Architecture Search designs and alters the internal layer structure and connectivity of the neural network itself to find optimal balances between model accuracy and hardware latency.
Prefix caching should be used when workloads have repeated prompt structures, such as system prompts in multi-turn chat applications or repeated queries against a single long document. Speculative decoding should be used to accelerate the token generation phase itself across general text generation tasks by using a small draft model to propose tokens that a larger base model verifies in parallel.
Vertex AI calculates custom serving quota against active, real-time compute resource utilization rather than the maximum replica limit (maxReplicaCount). While this allows unreserved quota to be shared efficiently across other workloads, a deployment cannot scale up to its configured maxReplicaCount if the overall project quota is exhausted when demand spikes.
Professional Machine Learning Engineer
Prepare and test your skills
Prepare and test your skills