Deploying Batch and Online Inference Models

Cost-optimized, high-throughput batch prediction is asynchronous distributed inference over large datasets without paying for always-on serving. Vertex AI Batch Prediction (opens in a new tab) provisions a temporary pool of compute nodes, spreads input instances across prediction containers in parallel, and writes all predictions to Cloud Storage or BigQuery when the job finishes. Throughput grows with node count, machine size, and request batching, while cost is controlled with preemptible capacity, reservations, and right-sized accelerators.

1.1Asynchronous Batch Job Architecture and Execution Flow

Vertex AI Batch Prediction is an asynchronous job service where data flows from a Cloud Storage or BigQuery source through a pool of prediction containers to a Cloud Storage directory or BigQuery table. The inference container runs as the user-provided custom service account, while read and write operations for instances and results are performed by the Agent Platform service agent, which by default has access to BigQuery and Cloud Storage. This split separates code execution identity from data access identity, so the job depends on both identities having correct permissions. A batch job moves through an ordered lifecycle:

1. Submit: The request specifies the model, input source and format, output destination, starting replica count, machine type, and optional accelerators.
2. Provision and distribute: The service provisions the requested compute nodes and distributes input instances across containers in parallel.
3. Predict: Each container receives groups of instances defined by the batch size and returns predictions.
4. Persist: The service agent writes the paired instances and predictions to the specified output directory or table, and the job transitions to complete.

1.2Input Sources and Result Destinations

Batch prediction input sources define where workers read data from and what format they must parse. The service accepts Cloud Storage files and BigQuery tables, and the output format does not need to match the input format, so a BigQuery table input can produce CSV objects in Cloud Storage. Supported Cloud Storage inputs include:

• jsonl for JSON Lines instances
• csv for comma-separated rows
• tf-record for TFRecord files
• tf-record-gzip for compressed TFRecord files
• file-list for a list of Cloud Storage files in other formats, such as images

BigQuery input carries a strict dependency: when BigQuery is selected as input, BigQuery must also be selected as output with a Google-managed encryption key, and customer-managed encryption keys (CMEK) are not supported. Completed jobs store output as JSON Lines files named {gcs_path}/prediction.results-{file_number}-of-{number_of_files_generated}, where the file count is nondeterministic because inference is distributed. Each output line pairs one input with its result using an instance key containing the sent value or Cloud Storage URI and a prediction key containing the value returned by the container.

1.3Throughput Tuning with Compute Sizing and Batch Size

Throughput tuning for batch prediction controls how many instances move through the node pool per second by adjusting parallelism, per-node power, and per-request packing. Three compute fields (opens in a new tab) govern the tradeoff: startingReplicaCount sets the number of nodes, machineSpec sets the machine type and optional accelerator type and count, and batchSize sets the number of instances sent in each prediction request with a default of 64. Increasing batch size can raise throughput but can also cause request timeouts, so it must be tested against model latency. Published machine recommendations are only a starting point, and throughput for a specific model should be measured with the Finding Ideal Machine Type notebook before scaling out.

GPU- and TPU-accelerated jobs may also need more CPUs and GPUs than online serving for work such as data preprocessing inside the container. Because startup alone can consume 10 minutes, targeting at least 20 minutes of total job time ensures a reasonable proportion of time and cost is spent on inference rather than provisioning.

1.4Cost-Optimized Capacity with Spot VMs and Dynamic Scheduling

Cost-optimized capacity matches the provisioning model to how predictable and fault-tolerant the batch workload is. Spot VMs are preemptible VMs that lower compute cost for fault-tolerant work such as evaluation and experimentation, but the job must tolerate preemption and retry. Configuring Spot VMs, Flex-Start VMs, or shared reservations causes the model server to deploy to a user-facing Agent Platform endpoint in Preview, which improves observability and reduces the number of quotas to manage. Capacity choices divide by duration and obtainability:

• Committed use discounts (CUDs): Apply to VMs and accelerators for stable 24x7 inference or training with fixed or predictable capacity requirements.
• Spot VMs: Apply to GKE nodes and Compute Engine VMs for fault-tolerant batch and evaluation tasks where preemption is acceptable in exchange for lower cost.
• Dynamic Workload Scheduler (DWS) Flex-Start mode: Applies to complex batch training that needs high-end GPUs for up to seven days with better obtainability than Spot alone.
• DWS Calendar mode: Applies to longer workloads up to three months that reserve specific GPUs such as H100 and H200 and TPUs such as Trillium.

For accelerator-heavy pipelines, Ray on Agent Platform (opens in a new tab) abstracts distributed infrastructure so applications request resources for training and serving without directly managing VMs and containers. Cluster Director manages accelerator groups with colocation and scheduling for multi-GPU training on A3 Ultra H200 and A4 B200 across GKE and Slurm clusters.

1.5Guardrails for Cost Control: Explanations, Feature Filtering, and Monitoring

Cost guardrails for batch prediction prevent expensive fan-out and unwanted data movement before the job scales to millions of instances. Feature-based explanations from Vertex Explainable AI should not run on large datasets because each input can fan out to thousands of requests based on possible feature values, which massively increases processing time and cost, and example-based explanations are not supported for batch inference at all. Feature filtering with instanceConfig reduces payload and cost by sending only needed fields, using either excludedFields such as customerId or includedFields such as col1 and col2 with instanceType set to object. Optional Model Monitoring analysis (opens in a new tab) for batch predictions in Preview adds skew detection against a training data source with alert thresholds, notification emails, and Cloud Monitoring channels for PagerDuty, Slack, and Pub/Sub.

Low-latency online inference is the architecture for serving real-time prediction requests behind autoscaled, accelerator-backed endpoints that meet strict latency and availability Service Level Objectives (SLOs). On Google Cloud, Gemini Enterprise Agent Platform (Agent Platform), Cloud Run, and Google Kubernetes Engine (GKE) (opens in a new tab) divide this work by operational control: Agent Platform provides managed training and serving with exported monitoring metrics, Cloud Run provides serverless containers for low-GPU and variable traffic, and GKE provides container orchestration with direct GPU and TPU control for generative AI and large language model (LLM) inference.

2.1Selecting Compute Platforms for Real-Time Serving

Platform selection for real-time serving starts from accelerator need, traffic shape, and control requirements. Route requests with low GPU needs to Cloud Run for cost and management efficiency and route complex, GPU-intensive tasks to GKE where custom runtimes, vLLM or Triton servers, and GPU and TPU pools can be tuned. Front diverse platforms with a centralized load balancer and use Cloud Service Mesh for inter-service communication and policy management, with consistent logging through Cloud Logging and metrics through Cloud Monitoring. For global users who need low latency, place a global external Application Load Balancer in front of regional inference deployments so traffic flows from the global edge to the nearest healthy backend.

The platforms can compose into one serving path: the global load balancer receives user traffic, the centralized routing layer sends light requests to Cloud Run and heavy generative requests to GKE, and the mesh enforces policy as traffic flows between services.

2.2Configuring Hardware Acceleration with Custom Containers

Hardware acceleration on GKE means running custom inference containers on nodes equipped with NVIDIA GPUs such as H100, A100, and L4, or with TPUs including Trillium, so prefill and decode execute on accelerators rather than CPUs. The inference Pod depends on the node pool for accelerator type, on the serving framework for execution, and on the storage mount for model weights; if any of those dependencies is unavailable or slow, first-token and end-to-end latency increase. To handle fluctuating capacity and performance needs, run a vLLM engine (opens in a new tab) that dynamically uses TPUs and GPUs for fungibility between accelerator types. For complex multi-GPU topology needs, use Cluster Director to deploy accelerator groups with colocation and scheduling for A3 Ultra H200 and A4 B200, and use Ray on Agent Platform to let applications request distributed resources without direct management of VMs and containers.

2.3Distributing Traffic with Model-Aware Routing

GKE Inference Gateway is the GKE load-balancing layer that deploys models behind a load balancer with model-aware routing. The gateway receives inference traffic and prioritizes instances with GPU and TPU accelerators for compute-intensive generative AI and LLM work, so heavy requests flow to accelerator-backed Pods while health determines eligibility. Configure detailed health checks to assess model status, not just Pod liveness, and expose LLM metrics from serving frameworks such as vLLM or NVIDIA Triton Inference Server. Integrate those metrics into Cloud Monitoring by using Google Cloud Managed Service for Prometheus, so routing, autoscaling, and alerting all depend on the same model signals.

To keep critical inference on suitable machines when GPU and TPU availability is constrained, use GKE custom compute classes to define a hierarchy of compute profiles with fallback policies for autoscaling. A profile can specify higher priority for reserved GPU or TPU instances and then fall back to cost-efficient Spot VMs if reserved resources are temporarily unavailable. Track canary releases and rollbacks by integrating Cloud Deploy with Cloud Monitoring, so a new model version receives a controlled share of traffic first and only then promotes to full traffic after latency and error signals remain healthy.

2.4Ensuring Autoscaling Capacity and Accelerator Obtainability

Autoscaling for low latency depends on accelerator obtainability, because a scaler cannot add replicas if GPUs or TPUs are unavailable in the zone. For stable 24x7 inference with fixed or predictable capacity, use committed use discounts (CUDs) for VMs and accelerators to reserve capacity at lower cost. For fault-tolerant evaluation and experimentation, use Spot VMs, which can be preempted but reduce cost, and never place strict-latency online serving solely on preemptible capacity without a reserved fallback.

For high-demand accelerators, use Dynamic Workload Scheduler (DWS) to persist the GPU request and provision capacity when available:

• Flex-Start mode supports dynamically provisioned resources for up to seven days with short-term reservations and pay-as-you-go billing, with discounts up to 53% for vCPUs, GPUs, TPUs, and attached Local SSD disks.
• Calendar mode reserves specific GPUs such as H100 and H200 and TPUs such as Trillium for longer-running workloads up to three months.

Monitor scaling behavior to ensure that autoscaling responds appropriately to traffic changes, and use Active Assist recommendations to detect unhealthy infrastructure that would block scale-out.

2.5Meeting Latency and Availability SLOs

SLOs for online inference translate business needs into measurable thresholds for success rate, latency, and output safety. Aligning SLOs directly with business needs focuses reliability work on the user-visible behavior that matters most. Representative production thresholds (opens in a new tab) from the documentation are:

• 99.9% of API calls must return a successful response.
• 95th percentile inference latency must be below 300 ms.
• Time to First Token (TTFT) must be below 500 ms for 99% of requests.
• Rate of harmful output must be below 0.1%.

These SLOs create tradeoffs: lowering concurrency per replica can reduce queueing and TTFT but requires more replicas and accelerators, while packing more concurrent requests improves throughput and cost but increases tail latency and saturation risk. Capacity planning must therefore combine historical utilization analysis in Cloud Monitoring, regular load testing to find bottlenecks, and proactive quota requests for Compute Engine, Agent Platform, and Cloud Storage justified by forecasting data.

2.6Measuring Inference Performance with Latency and Throughput Metrics

Inference performance metrics separate the prefill phase that produces the first token from the decode phase that streams the remaining tokens. Time to First Token (TTFT) is the time in milliseconds to generate the first token, request latency is the end-to-end time to complete a request, and token metrics decompose that total. Normalized Time Per Output Token (NTPOT) is measured as request_latency / total_output_tokens, while Time Per Output Token (TPOT) is measured as (request_latency - time_to_first_token) / (total_output_tokens - 1). Inter-token latency (ITL) measures the time between each individual output token generation and is then aggregated into mean, median, and percentile values such as p90, which reveals streaming stalls that averages hide.

Throughput metrics measure volume over time and must be interpreted with context length, because requests per second alone is unreliable for LLMs with widely varying contexts. Output tokens per second is measured as total_output_tokens_generated_by_server / elapsed_time_in_seconds, input tokens per second uses total input tokens over the same elapsed time, and tokens per second counts both input and output tokens to compare workloads with high prefill versus high decode times. Use the model observability dashboard to view QPS, token throughput, and first-token latencies collected by Cloud Monitoring.

2.7Monitoring Latency, Traffic, Errors, and Saturation

Infrastructure and application monitoring for AI serving tracks processor usage for CPU, GPU, and TPU, plus memory usage, network throughput and latency, and disk I/O across Agent Platform and self-managed resources such as GKE nodes and Cloud Run instances. Organize that telemetry around the four golden signals, where saturation shows how close accelerators and memory are to capacity limits and therefore predicts queueing before latency SLOs breach:

• Latency: Time to respond to requests, including TTFT and end-to-end request latency.
• Traffic: Volume of requests or workload, including QPS and token throughput.
• Error rate: Rate of failed requests or operations, including model-specific failures.
• Saturation: Utilization of CPU, memory, and GPU or TPU accelerators.

Collect, store, and visualize metrics with Cloud Monitoring using pre-built dashboards and custom dashboards for training, endpoints, and performance, and create custom metrics and alerts integrated with incident tools such as PagerDuty or Slack. Collect detailed logs with Cloud Logging for troubleshooting, pinpoint request flows across distributed inference microservices with Cloud Trace, identify hot function blocks with Cloud Profiler, and gather per-process GPU utilization, memory, and temperature with NVIDIA Data Center GPU Manager (DCGM). For generative quality, continuously monitor toxicity scores, user-feedback ratings, hallucination or query-resolution failures, and for retrieval-augmented generation, the relevance of retrieved context and groundedness of responses.

2.8Accelerating Scale-Out with Model Weight Storage

Model weight loading controls cold start and scale-out latency, because a new replica cannot serve until weights move from storage to the accelerator host. Cloud Storage FUSE mounts model weights from Cloud Storage with caching and parallel downloads and fits deployments with limited node scaling. When many Pods must start together, the shared file path becomes the bottleneck and a higher-throughput mount is required.

Data flows from the weight store through the mount to each inference Pod, so the storage service sits on the critical path of every autoscaling event. Choosing Lustre or Hyperdisk ML over a basic mount trades operational and cost overhead for predictable startup time when dozens or thousands of accelerator Pods need the same large weights at once.

Deploying generative models and intelligent agents on Google Cloud means choosing where the foundation model runs, how it is served at scale, and how it is grounded with enterprise data. Vertex AI provides fully managed infrastructure with first-class generative APIs for Gemini and task-specific models, Vertex AI Model Garden (opens in a new tab) provides one-click discovery and deployable third-party and open-source models, and self-hosted targets such as Google Kubernetes Engine (GKE), Cloud Run, and Compute Engine provide full control over kernels, operating systems, and runtimes. Agents built on these models combine a served model with grounding and retrieval, tuning, and governance so responses stay accurate, safe, and observable in production.

3.1Foundation Model Selection and Deployment Targets

Foundation model selection starts from the modality and task, then maps to a serving target based on control and operational needs. Gemini serves text generation and multimodal prompts that include images or video, Codey serves code generation, Imagen serves image generation, and text-embedding models serve search, classification, and clustering. Google Cloud exposes flagship Gemini models directly through Vertex AI APIs, while third-party and open-source models such as DeepSeek-V3 or Llama can be deployed to Vertex AI Model Garden or self-hosted on GKE, Cloud Run, or Compute Engine. Choose a managed Vertex AI API when no custom serving container is needed, choose Model Garden when an open model needs enterprise deployment with managed endpoints and governance, and choose Compute Engine when the model requires a specialized kernel, legacy OS, or special licensing terms that managed runtimes do not allow.

The deployment targets differ by management, customization, and scaling model:

Traffic flows from the client application to the selected target, and the target depends on Artifact Registry for containers, Cloud Storage for weights, and Identity and Access Management for invocation rights. For high-performance training and inference of large language models (LLMs) and diffusion models, configure accelerators such as GPUs or Cloud Tensor Processing Units (TPUs) behind the serving target.

3.2Optimized Serving With vLLM Containers on Vertex AI

Optimized serving with vLLM packages an open model with a high-throughput inference server that uses continuous batching, chunked prefill, and paged key-value cache management. The Model Garden vLLM container image, such as pytorch-vllm-serve, starts a Ray launcher that runs vllm.entrypoints.api_server on port 7080 for the model identified by the MODEL_ID environment variable. Parallelism and memory flags control how the model shards across accelerators, including --tensor-parallel-size, --pipeline-parallel-size, --gpu-memory-utilization, --max-model-len, --max-num-seqs, --enable-chunked-prefill, and --kv-cache-dtype, plus --trust-remote-code for models with custom code. This container lets a very large model such as deepseek-ai/DeepSeek-V3 serve OpenAI-compatible chat completions from Vertex AI while retaining vLLM-level throughput tuning.

Deploying a large vLLM model to Vertex AI follows an ordered workflow:

1. Upload model: Upload the model to the Model Registry with gcloud ai models upload, setting the vLLM container image, container args, container port 7080, extended container deployment timeout such as 7200 seconds, and MODEL_ID env var.
2. Create dedicated endpoint: Create a Vertex AI Endpoint with dedicatedEndpointEnabled set to true for chat-completion traffic.
3. Deploy model: Deploy the uploaded model to the endpoint with a GPU machine type such as a3-highgpu-8g, accelerator nvidia-h100-80gb with count 8, multihost-gpu-node-count for multi-node sharding, and traffic split 0=100.
4. Poll operation: Poll the long-running deploy operation with gcloud ai operations describe until the response includes "done": true, because large-model deployment continues in the background after a CLI timeout.

3.3Dedicated Endpoints and Multi-GPU Deployment for Large Models

Vertex AI dedicated endpoints are single-tenant online inference endpoints required to support chat-completion requests from Model Garden generative containers. A standard shared endpoint accepts general prediction traffic, while a dedicated endpoint isolates the deployment and exposes the chat-completion path the vLLM server expects. The endpoint must be created before deployment with a REST POST to /v1/projects/PROJECT_ID/locations/REGION/endpoints containing displayName and "dedicatedEndpointEnabled": true. The deployment then binds a specific model version to that endpoint, so the endpoint depends on both the Model Registry artifact and the accelerator pool.

Large-model deployment adds multi-host GPU topology (opens in a new tab) and extended lifecycle handling. The deploy call specifies the source model ID from gcloud ai models list, the destination endpoint ID from gcloud ai endpoints list, machine type, accelerator type and count, and multihost node count such as 2 nodes of 8x H100 for DeepSeek-V3 with tensor-parallel-size 8 and pipeline-parallel-size 2. First the control plane provisions accelerators and pulls the container, then it loads sharded weights and starts the vLLM server, and only after health checks pass does the endpoint move to ready and receive traffic. If the client times out, the server-side operation remains running until it transitions to done or failed, so automation must poll the operation ID rather than retrying the deploy.

3.4Grounding and Retrieval-Augmented Generation for Agents

Grounding is the practice of connecting generation to verifiable real-time data so responses are informed and accurate, and retrieval-augmented generation (RAG) is the pattern that implements it. Data flows from source systems to an index, then from the index to the prompt: the agent retrieves relevant passages or rows for a query, inserts them into the model prompt, and the model generates a response conditioned on both the query and the retrieved evidence. Gemini models can evaluate whether internal knowledge is sufficient or whether grounding with Google Search is required for up-to-date internet information. Embeddings for similarity search are commonly stored in a vector database, but grounding can also query a non-vector database such as Cloud SQL or Firestore and place the query result directly in the prompt.

The retrieval layer differs by how much infrastructure and embedding control the team retains:

When none of those managed options fit, use a reference architecture to build a tailor-made search engine and vector database, with BigQuery or Cloud Storage as the evaluation and corpus source. The agent depends on the retriever for factual context and the retriever depends on the data source for freshness, so stale indexes or missing access boundaries directly degrade answer quality and governance.

3.5Tuning, Evaluation, and Governance for Production Agents

Production governance for generative models and agents combines tuning, standardized evaluation, and lifecycle monitoring. Vertex AI Model Garden provides pretrained foundation models for text, image, and code that can be used as-is or fine-tuned for a specific domain, with high-performance GPU or TPU training for LLMs and diffusion models. Prompt engineering, response filtering, and human-in-the-loop review are part of the generative lifecycle alongside data augmentation and synthetic data to improve prompt diversity and robustness. The tuned artifact is stored as a versioned model, so the endpoint always serves a traceable model version rather than an untracked checkpoint.

Evaluation and monitoring close the loop from experiment to production operation:

• Gen AI evaluation service evaluates generative models and applications against custom judgment criteria and supports prompt engineering, RAG quality, and agent optimization.
• Vertex AI evaluation service provides ready-to-use primitives to track model performance as part of the ML lifecycle.
• Automatic side-by-side (AutoSxS) provides pairwise model-based comparison for ranking candidate outputs or models.
• Model Registry on Vertex AI stores, organizes, and versions model binaries and metadata for repeatable deployment.
• Model Monitoring on Vertex AI detects data drift, tracks performance, and flags anomalies, including shifts in generative output quality and safety compliance for production agents.

Data flows from BigQuery or Cloud Storage evaluation datasets into these evaluation services, and monitoring signals flow back to retraining, re-tuning, or prompt-revision decisions. Version control for code in Git, containers in Artifact Registry, features in Feature Store (opens in a new tab), and lineage in Vertex ML Metadata keeps experiments reproducible and keeps experimental projects isolated from production projects, networks, and data.

Key Facts

• Vertex AI Batch Prediction runs asynchronous jobs where data flows from Cloud Storage or BigQuery through prediction containers to a Cloud Storage directory or BigQuery table, with a default batchSize of 64.
• BigQuery as batch prediction input requires BigQuery as output with Google-managed encryption, and customer-managed encryption keys are not supported for that combination.
• GPU- and TPU-accelerated batch jobs incur about 10 minutes of accelerator startup time, so jobs should target at least 20 minutes to keep most cost on inference.
• GKE Inference Gateway routes heavy generative AI traffic to GPU- and TPU-backed Pods using model-aware routing, health checks, and Prometheus metrics, with GKE custom compute classes providing fallback from reserved accelerators to Spot VMs.
• Vertex AI dedicated endpoints are single-tenant endpoints required for Model Garden vLLM chat completions and must be created via REST API with dedicatedEndpointEnabled set to true.