Serving and Scaling Models on Vertex AI

1.1Batch and Online Inference Architectures

Batch and online inference are two distinct serving patterns for different latency and throughput goals. Vertex AI Batch Prediction (opens in a new tab) is an asynchronous service where data flows from a Cloud Storage or BigQuery source through a pool of prediction containers to a Cloud Storage directory or BigQuery table. The inference container runs as the user-provided custom service account, while read and write operations for instances and results are performed by the Agent Platform service agent. Online serving instead handles real-time requests behind autoscaled endpoints on Agent Platform, Cloud Run, and Google Kubernetes Engine (GKE), optimizing for latency, availability, and accelerator obtainability. Batch prediction moves through an ordered lifecycle where each stage depends on the previous one:

1. Submit: The request specifies the model, input source and format, output destination, starting replica count, machine type, and optional accelerators.
2. Provision and distribute: The service provisions the requested compute nodes and distributes input instances across containers in parallel.
3. Predict: Each container receives groups of instances defined by batch size and returns predictions.
4. Persist: The service agent writes paired instances and predictions to the output destination, and the job transitions to complete.

1.2Batch Input Formats, Outputs, and Cost Tuning

Batch prediction input sources define where workers read data from and what format they must parse. The service accepts Cloud Storage files in jsonl, csv, tf-record, tf-record-gzip, and file-list formats, plus BigQuery tables, and the output format does not need to match the input format. Completed jobs store output as JSON Lines files named prediction.results-{file_number}-of-{number_of_files_generated}, where file count is nondeterministic because inference is distributed. Each output line pairs one input with its result using an instance key and a prediction key. Throughput tuning controls how many instances move through the node pool per second through startingReplicaCount for node count, machineSpec for machine type and accelerator type and count, and batchSize for instances per request with a default of 64. Increasing batch size can raise throughput but can cause request timeouts, so it must be tested against model latency. GPU- and TPU-accelerated jobs incur about 10 minutes of accelerator startup time, so jobs should target at least 20 minutes of total time to keep most cost on inference. Cost-optimized capacity uses committed use discounts for stable 24x7 inference, Spot VMs for fault-tolerant work that can tolerate preemption, Dynamic Workload Scheduler Flex-Start for up to seven days of high-end GPU obtainability, and Calendar mode for reservations up to three months.

1.3Online Serving Platforms and Generative Endpoints

Low-latency online inference serves real-time requests behind autoscaled, accelerator-backed endpoints. Traffic flows from a global external Application Load Balancer to the nearest healthy regional backend, with light requests routed to Cloud Run and heavy generative requests routed to GKE. Cloud Service Mesh manages inter-service communication and policy as traffic flows between services, while GKE Inference Gateway prioritizes GPU- and TPU-backed Pods for compute-intensive generative work based on health and model metrics from vLLM or NVIDIA Triton Inference Server.

Generative model deployment maps modality and task to a serving target, where Gemini serves text and multimodal prompts, Codey serves code, Imagen serves images, and text-embedding models serve search. Model Garden (opens in a new tab) deploys third-party and open models such as DeepSeek-V3 or Llama with managed endpoints, while GKE, Cloud Run, or Compute Engine self-host models needing custom runtimes. Large vLLM models require dedicated endpoints, which are single-tenant endpoints created with a REST call containing dedicatedEndpointEnabled: true before model deployment binds a specific model version. Deployment specifies machine type, accelerator type and count, and multihost node count, such as 2 nodes of 8x H100 for DeepSeek-V3, and the endpoint moves to ready only after accelerators provision, weights load, and health checks pass.

1.4Grounding, Metrics, and Monitoring for Inference

Grounding connects generation to verifiable data through retrieval-augmented generation (RAG). Data flows from source systems to an index and then to the prompt: the agent retrieves relevant passages, inserts them into the model prompt, and the model generates a response conditioned on both query and evidence. Vertex AI Search provides fully managed search without direct embedding access, Vertex AI RAG Engine provides managed chunking and retrieval wiring, and Vertex AI Vector Search provides low-latency similarity search with full index control. Inference performance separates the prefill phase from the decode phase, where Time to First Token (TTFT) measures responsiveness and Time Per Output Token (TPOT) measures average decode cost as (request_latency - time_to_first_token) / (total_output_tokens - 1). Requests per second alone does not compare large language model (LLM) workloads with different context lengths, while inter-token latency p90 reveals streaming stalls. Operational monitoring organizes telemetry into latency, traffic, error rate, and saturation of CPU, GPU, TPU, and memory, where saturation predicts queueing before latency objectives breach.

1.5Framework Packaging with Prebuilt and Custom Containers

Packaging for multi-framework serving matches each model to the right container build and protocol so PyTorch, XGBoost, TensorFlow, and scikit-learn run side by side. Vertex AI Prediction flows model artifacts from Cloud Storage to a Model Registry Model resource and then to a Vertex AI Endpoint that hosts compute and routes requests. Prebuilt containers bundle a specific framework version with CPU or GPU dependencies, and the serving container version must match the framework version used for training. Custom containers package model artifacts, dependencies, and a custom prediction server for deployment on Vertex AI Prediction, GKE, or Cloud Run.

Exported artifacts must comply with exact filename contracts, where TensorFlow uses a SavedModel directory with saved_model.pb plus variables and assets, PyTorch uses model.mar with model-name model, XGBoost uses model.bst or model.joblib, and scikit-learn uses model.joblib. Custom exports without Keras or an Estimator must use the serve tag and serving_default signature, or Vertex AI Prediction cannot discover the serving graph. Custom prediction routines extend a prebuilt image with handler code so traffic flows from the Endpoint to the custom preprocessor, then to the framework predict call, then to the custom postprocessor. Deployment follows an ordered workflow:

1. Save locally: Export the SavedModel or framework artifact to a local directory such as ./model/.
2. Stage to Cloud Storage: Copy every artifact file to the serving bucket directory that will become artifact_uri.
3. Upload Model: Call aiplatform.Model.upload with display name, artifact_uri, and serving container image URI.
4. Create Endpoint: Call aiplatform.Endpoint.create with a display name to create the logical serving endpoint.
5. Deploy model: Call model.deploy with endpoint, deployed model display name, and machine type such as n1-standard-2.

1.6Model Registry, Versions, and Governance

Model Registry (opens in a new tab) is the central repository for model versions and metadata, while Artifact Registry is the immutable store for underlying files and containers. Each Model Registry version points to tagged artifacts plus runtime dependencies, container specification, and input-output schemas, and only models that pass validation are registered for production. Cloud Storage holds validated configuration files and validation artifacts for a durable audit trail, and artifacts must be staged in a sandboxed serving environment to verify load and invocation before promotion.

Semantic version aliasing attaches a movable pointer such as dev, staging, production, champion, or challenger to an immutable version ID such as v3. Agent code references the alias, so resolving production returns whichever version currently holds that alias without changing the endpoint URI. Promotion reassigns aliases from development to staging to production only after release criteria are met, and superseded versions are retained with labels and access controls for audit. Evaluation metadata links each version to training parameters, metrics, dataset version, foundation-model details, and container tag, while Identity and Access Management (IAM), Cloud Audit Logs, and Vertex ML Metadata enforce least privilege, traceability, and versioning.

1.7Rollout Strategies and Traffic Management

Controlled rollout strategies compare model versions on live traffic while limiting risk and preserving fast rollback. Gemini Enterprise Agent Platform decouples the endpoint layer from the models behind it, so one stable endpoint URL can route to multiple DeployedModel resources. An endpoint owns a traffic-split map, so traffic flows from the agent client to the endpoint URI to the selected deployed model, and every split on one Endpoint must total exactly 100. A progressive canary follows an ordered sequence:

1. Deploy side-by-side: Deploy the new version alongside the stable version without shifting traffic yet.
2. Shift initial share: Shift a small share first, such as 1% to the new version.
3. Validate health and success criteria: Hold at that share while comparing error rate, latency, and prediction metrics.
4. Increase in bounded steps: Increase the canary share only after the current step succeeds, for example by growing the shifted share by 50% every 5 minutes.
5. Complete or roll back: Complete cutover to 100% when all stages pass, or return traffic to the stable version on degradation.

A/B testing deploys two versions to the same endpoint and splits traffic, such as 20% to new and 80% to old, which avoids breaking changes and sudden cutover. Because compute is associated with the DeployedModel rather than the Endpoint, each variant carries its own machine type, accelerator, and replica counts with autoscaling on concurrent requests. Ordered rollouts across fleets use the mesh-modernization-order label with early, default, and late waves, where each cluster group modernizes sequentially and waits for automated steps to complete before beginning the next group.

1.8Preprocessing, Postprocessing, and Streaming Enrichment

Custom prediction routines and fully custom containers both run on Vertex AI endpoints but divide HTTP server ownership differently. A custom prediction routine keeps the prebuilt server for liveness, health, and routing and adds a custom predictor handler, while a custom container requires the team to build and operate the entire HTTP server. Custom code sits on the request path between Vertex AI and the model: traffic flows from the endpoint to the container, the handler transforms the payload, the model produces raw outputs, and the handler transforms those outputs back into the client response. The container moves through an ordered serving lifecycle:

1. Container start: The startup command from Dockerfile ENTRYPOINT, CMD, or containerSpec overrides launches the HTTP server in a running state indefinitely.
2. Liveness checks: Vertex AI sends repeated liveness probes, and the server must respond so the platform does not restart the container.
3. Health checks: Only after the server is live, Vertex AI sends health probes to verify artifacts and dependencies, moving the container from starting to healthy.
4. Inference requests: Only after the container is healthy, Vertex AI forwards prediction traffic where the handler deserializes, preprocesses, invokes the model, postprocesses, and returns a response.

In-artifact transformations embed preprocessing directly into the served artifact so the same code runs in training and serving and avoids skew. Low-latency feature enrichment joins live events with precomputed features, where data flows from Cloud Pub/Sub to a Dataflow Apache Beam pipeline that calls Vertex AI Feature Store with the entity key and then calls the endpoint with RunInference. The end-to-end flow runs in strict order: producers publish events to Pub/Sub, Dataflow reads the stream, enrichment joins Feature Store values onto each event, RunInference scores the enriched record, and the pipeline forwards predictions for storage or alerting.

2.1Feature Store Storage and Serving Pipeline

Agent Platform Feature Store is a managed service that centralizes machine learning features in BigQuery for offline training and in a FeatureOnlineStore for low-latency online inference. Data flows from batch and streaming sources into BigQuery tables or views, then from BigQuery into the online store through FeatureView sync operations, then from the online store to real-time applications. BigQuery remains the durable source of truth with full history, while the online store holds materialized latest values optimized for serving. Feature storage architecture uses BigQuery tables or views as the authoritative source behind every feature view, with structured data stored as materialized tables for read speed and read through the BigQuery Storage API. Feature groups and features catalog those sources, a FeatureOnlineStore instance provides serving capacity, and FeatureView resources select the columns and rows needed for serving. Unstructured inputs such as images, video, and audio remain in Cloud Storage in large sharded containers with at least 100 MB per file and 100 to 10,000 shards. Batch ingestion uses Dataflow with the BigQuery I/O connector, while streaming updates land in BigQuery first and then refresh the online store.

2.2Feature Views and Sync Modes for Consistency

A FeatureView is the materialized serving definition that selects data from BigQuery and controls how it loads into the FeatureOnlineStore. During online serving, requests flow from the application to the online store endpoint, which returns latest values by entity ID without querying BigQuery. During offline serving, training and batch inference read historical rows directly from BigQuery, preserving history the online store does not retain. Point-in-time correctness uses a timestamp column with ML.FEATURES_AT_TIME or ML.ENTITY_FEATURES_AT_TIME so training never sees future data. Online serving is configured as Bigtable online serving or Optimized online serving, where Bigtable hosts features for high-throughput key lookups and Optimized exposes a dedicated endpoint that can support vector similarity search for approximate nearest neighbors.

2.3Request Batching, Concurrency, and Governance

Request batching and concurrency tuning controls how lookup requests flow from Spanner to Agent Platform to minimize latency and avoid overload. The default_batch_size setting packs rows into each API request, and smaller batches reduce token and byte size for large payloads such as text fields. For GoogleSQL, an oversized batch returns INVALID_ARGUMENT with "Request is too large", which is fixed by reducing default_batch_size to split work across more requests. The @{remote_udf_max_outstanding_rpcs} hint controls concurrent requests from Spanner to Agent Platform and overrides default_batch_size, where raising it increases parallelism but also increases Spanner CPU and memory usage. Governance ensures served features are valid, fresh, traceable, and accessible only to authorized callers across the path from BigQuery through feature views to the online store. Validation uses Dataflow, Managed Service for Apache Spark, or BigQuery SQL checks plus TensorFlow Data Validation (TFDV) for schema and anomaly detection, with materialized tables preferred over views for consistency. Monitoring distinguishes training-serving skew from inference drift and staleness, where a Failed sync is treated as a freshness incident. Each feature view reads BigQuery with a service account granted the BigQuery Data Viewer role, with IAM boundaries, audit logging, Vertex ML Metadata, and Model Registry preserving access control and lineage.

2.4Public and Private Endpoints for Online Serving

Vertex AI online serving delivers predictions through a Vertex AI Endpoint that routes traffic to one or more DeployedModels. The serving path flows from model artifacts in Google Cloud Storage (GCS) to a Vertex AI Model in the Model Registry, then to the endpoint that owns routing and scaling. Each deployed model owns its container, machine type, accelerator, and traffic share, and deployment can take more than five minutes. Public endpoints expose scalable prediction over HTTPS with a stable address that persists while models behind it change. Registration binds artifacts under gs://BUCKET/DIRECTORY to a serving container, endpoint creation binds the endpoint to a project and region, and deployment attaches a Model ID to an endpoint ID with traffic splitting such as --traffic-split=0=100 for canary and zero-downtime updates. Private endpoints keep prediction traffic inside the private network through Private Service Connect (PSC), where requests flow from a client in a consumer Virtual Private Cloud (VPC) to a consumer forwarding rule to the producer ServiceAttachment and then to the Vertex AI endpoint. Access is gated by an immutable projectAllowlist, and internal DNS uses Cloud DNS private zones with A records to 199.36.153.4 through 199.36.153.7 with TTL 300. Attachment stability depends on active deployments: with no deployments for more than 10 minutes Vertex AI might delete and recreate the attachment, breaking the forwarding-rule binding.

2.5Hardware Selection for Training and Serving

Google Kubernetes Engine (GKE) (opens in a new tab) accelerator-optimized families span A4X Max, A4X, A4, A3 Ultra, A3 Mega, A3 High, A3 Edge, A2, G4, G2, and N1 with T4, complemented by Cloud TPUs and CPU platforms. Throughput, latency, billing rules, and discount limits together determine total cost for training versus serving on GKE and Compute Engine. Cloud TPUs accelerate large matrix operations with on-chip high-bandwidth memory (HBM) and connect in groups called slices that scale with little to no code changes.

Billing for accelerator-optimized machines charges for attached GPUs, predefined vCPU, memory, and bundled Local SSD where applicable. A resource is eligible for only one discount at a time, so on-demand, Spot, Flex-start, and reservation discounts do not stack. Flexible committed use discounts (CUDs) for N1 with GPUs cover only vCPUs, memory, and Local SSD, while for G2 and G4 they cover vCPUs, memory, GPUs, and Local SSD. Spot VMs offer 61% to 90% discounts with preemption at any time, while Flex-start up to seven days and Calendar mode up to 90 days offer up to 53% discounts with dense allocation. Inference signals separate responsiveness from efficiency, where TTFT, TPOT, Normalized Time Per Output Token (NTPOT), inter-token latency, and end-to-end latency measure latency, and requests, output, input, and total tokens per second measure throughput.

2.6Scaling Backends with Autoscaling and Provisioned Throughput

Scalable serving backends dynamically adjust containerized and managed inference capacity to handle fluctuating throughput without breaching latency objectives. Autoscaling signals combine latency, traffic, errors, and saturation of CPU, memory, and GPU or TPU with token-aware metrics such as model queries per second (QPS), token throughput, and first-token latencies from vLLM and Triton through Managed Service for Prometheus into Cloud Monitoring. Cold starts force new replicas to load multi-gigabyte weights before serving, so minimum replicas preserve warm capacity, maximum replicas cap cost and quota, and scale-to-zero saves idle cost but forces full initialization.

Provisioned Throughput (opens in a new tab) reserves deterministic throughput for production generative AI with weekly or monthly cost, while Standard pay-as-you-go uses shared regional capacity without manual quota requests. Quotas define when the backend returns error 429 Resource Exhausted for queries-per-minute or token limits, such as 10 QPM for Gemini 2.5 Pro under Standard. Custom container requirements for DeployedModel scale replicas between dedicatedResources.minReplicaCount and maxReplicaCount toward a 60% utilization target on CPU, or the higher of CPU or GPU when accelerators are present. High-performance runtimes such as vLLM, Triton, TorchServe, and Ray Serve raise throughput with concurrency only to the inflection point where throughput stops increasing and latency climbs sharply.

2.7Tuning Models for Training and Serving in Production

Tuning for production balances predictive performance against inference latency, throughput, and infrastructure cost. Agent Platform Vizier suggests parameter combinations as an independent black-box optimizer, training workers run trials and report measurements, and serving infrastructure is then sized, scaled, and compiled for live traffic. A study configuration defines the problem, a study implements it through trials, a trial is one set of input values, and a measurement is the observed outcome. Black-box search behavior determines how efficiently the space is explored:

Online serving infrastructure sizes Vertex AI Prediction endpoints so live traffic gets fast answers without overspending, with a minimum of two nodes for high-availability so one failure does not remove capacity. Vertex AI load-balances across zones without cross-region routing, where public endpoints retry automatically after a zonal outage while private endpoints require client retry with exponential backoff. Hardware-aware compilation uses TensorRT-LLM for NVIDIA GPUs and vLLM with prefix caching, Cache Aware Routing, and speculative decoding to meet TTFT and tokens per second at saturation. Async buffering places Pub/Sub with 7 to 31 days retention before inference and scales subscribers and servers on backlog with dead-letter handling.

Key Facts

• Vertex AI Batch Prediction runs asynchronous jobs where data flows from Cloud Storage or BigQuery through prediction containers to a Cloud Storage directory or BigQuery table, with a default batch size of 64.
• GPU-accelerated batch jobs incur about 10 minutes of startup time, so jobs should target at least 20 minutes to keep most cost on inference.
• Dedicated endpoints are single-tenant endpoints required for Model Garden vLLM chat completions and must be created via REST with dedicatedEndpointEnabled set to true.
• Private Service Connect prediction traffic flows from a consumer VPC forwarding rule to a producer ServiceAttachment gated by an immutable projectAllowlist, with private DNS using A records to 199.36.153.4 through 199.36.153.7.
• Provisioned Throughput reserves deterministic throughput with weekly or monthly cost for high-throughput workloads, while Vertex AI Endpoints scale DeployedModel replicas between min and max toward 60% CPU or GPU utilization.