Deploying Models for Batch and Online Inference

Batch and online inference on Google Cloud separate large asynchronous jobs from real-time serving. Vertex AI Batch Prediction provisions temporary compute to score large datasets and writes results to storage, while Gemini Enterprise Agent Platform, Cloud Run (opens in a new tab), and Google Kubernetes Engine (GKE) serve live requests behind autoscaled endpoints. Batch optimizes for throughput and cost per prediction, and online optimizes for latency, availability, and accelerator obtainability. Generative models and agents add a third pattern where a foundation model from Model Garden is deployed to a dedicated endpoint and grounded with retrieved data.

1.1Asynchronous Batch Prediction Architecture and Lifecycle

Vertex AI Batch Prediction is an asynchronous job service where data flows from a Cloud Storage or BigQuery source through a pool of prediction containers to a Cloud Storage directory or BigQuery table. The inference container runs as the user-provided custom service account, while read and write operations for instances and results are performed by the Agent Platform service agent. This split separates code execution identity from data access identity, so the job depends on both identities having correct permissions. The job moves through an ordered lifecycle:

1. Submit: The request specifies the model, input source and format, output destination, starting replica count, machine type, and optional accelerators.
2. Provision and distribute: The service provisions the requested compute nodes and distributes input instances across containers in parallel.
3. Predict: Each container receives groups of instances defined by batch size and returns predictions.
4. Persist: The service agent writes paired instances and predictions to the output directory or table, and the job transitions to complete.

1.2Batch Input Formats and Output Destinations

Batch prediction input sources define where workers read data from and what format they must parse. The service accepts Cloud Storage files and BigQuery tables, and the output format does not need to match the input format. Supported Cloud Storage inputs include jsonl, csv, tf-record, tf-record-gzip, and file-list for lists of files such as images. Completed jobs store output as JSON Lines files named prediction.results-{file_number}-of-{number_of_files_generated}, where file count is nondeterministic because inference is distributed. Each output line pairs one input with its result using an instance key and a prediction key.

1.3Throughput and Cost Tuning for Batch Jobs

Throughput tuning for batch prediction controls how many instances move through the node pool per second by adjusting parallelism, per-node power, and per-request packing. Three fields govern the tradeoff: startingReplicaCount sets node count, machineSpec sets machine type and accelerator type and count, and batchSize sets instances per prediction request with a default of 64. Increasing batch size can raise throughput but can cause request timeouts, so it must be tested against model latency. GPU- and TPU-accelerated jobs incur about 10 minutes of accelerator startup time, so jobs should target at least 20 minutes of total time to keep most cost on inference rather than provisioning.

Cost-optimized capacity matches the provisioning model to how fault-tolerant the workload is. Committed use discounts apply to stable 24x7 inference with predictable capacity, while Spot VMs lower cost for fault-tolerant evaluation and experimentation that can tolerate preemption. Dynamic Workload Scheduler Flex-Start mode improves obtainability of high-end GPUs for up to seven days, and Calendar mode reserves specific GPUs and TPUs for up to three months. Feature-based explanations should not run on large datasets because each input can fan out to thousands of requests, and example-based explanations are not supported for batch inference at all.

1.4Low-Latency Online Serving on Agent Platform, Cloud Run, and GKE

Low-latency online inference serves real-time requests behind autoscaled, accelerator-backed endpoints that meet latency and availability objectives. Agent Platform provides managed training and serving with centralized monitoring, Cloud Run provides serverless containers for variable traffic with low GPU needs, and GKE provides Kubernetes orchestration with direct GPU and TPU control for generative and large language model (LLM) inference. Traffic flows from a global external Application Load Balancer to the nearest healthy regional backend, with light requests routed to Cloud Run and heavy generative requests routed to GKE. Cloud Service Mesh manages inter-service communication and policy as traffic flows between services.

GKE Inference Gateway (opens in a new tab) is the GKE load-balancing layer that prioritizes GPU- and TPU-backed Pods for compute-intensive generative work. The gateway receives inference traffic and routes heavy requests to accelerator-backed Pods based on health and model metrics from vLLM or NVIDIA Triton Inference Server. GKE custom compute classes define profiles with fallback policies, such as priority for reserved accelerators with fallback to Spot VMs when reserved capacity is unavailable. Model weight loading sits on the critical path of every scale-out event, so Cloud Storage FUSE fits limited scaling while Managed Lustre and Hyperdisk ML support thousands of Pods starting together.

1.5Generative Model Deployment with Dedicated Endpoints

Generative model deployment maps modality and task to a serving target based on control needs. Gemini serves text and multimodal prompts, Codey serves code generation, Imagen serves image generation, and text-embedding models serve search and classification. Vertex AI managed APIs serve first-party models without custom serving code, Vertex AI Model Garden deploys third-party and open models such as DeepSeek-V3 or Llama with managed endpoints, and GKE, Cloud Run, or Compute Engine self-host models that need custom runtimes or operating system control. Optimized serving with vLLM uses continuous batching, chunked prefill, and paged key-value cache management to serve OpenAI-compatible chat completions.

Large vLLM models on Vertex AI require dedicated endpoints, which are single-tenant endpoints that expose the chat-completion path. The endpoint must be created with a REST call containing dedicatedEndpointEnabled: true before model deployment, and deployment then binds a specific model version to that endpoint. A large-model deployment specifies machine type, accelerator type and count, and multihost node count, such as 2 nodes of 8x H100 for DeepSeek-V3. First the control plane provisions accelerators and pulls the container, then it loads sharded weights and starts the vLLM server, and only after health checks pass does the endpoint move to ready and receive traffic.

1.6Grounding, Metrics, and Monitoring for Production Inference

Grounding connects generation to verifiable data through retrieval-augmented generation (RAG). Data flows from source systems to an index and then to the prompt: the agent retrieves relevant passages, inserts them into the model prompt, and the model generates a response conditioned on both query and evidence. Vertex AI Search provides fully managed search without direct embedding access, Vertex AI RAG Engine provides managed chunking and retrieval wiring, and Vertex AI Vector Search provides low-latency similarity search with full index control. The agent depends on the retriever for factual context and the retriever depends on the data source for freshness.

Inference performance separates the prefill phase from the decode phase. Time to First Token (TTFT) measures responsiveness of the first token, Time Per Output Token (TPOT) measures average decode cost as (request_latency - time_to_first_token) / (total_output_tokens - 1), and inter-token latency p90 reveals streaming stalls. Requests per second alone does not compare LLM workloads with different context lengths. Operational monitoring organizes telemetry into latency, traffic, error rate, and saturation, where saturation of CPU, GPU, TPU, and memory predicts queueing before latency objectives breach. Representative thresholds include 99.9% successful responses, 95th percentile latency below 300 ms, and TTFT below 500 ms for 99% of requests.

Packaging and serving for multi-framework environments matches each model to the right container build, compute platform, and protocol so PyTorch, XGBoost, TensorFlow, and scikit-learn models run side by side. Vertex AI Prediction flows model artifacts from Cloud Storage to a Model Registry Model resource and then to a Vertex AI Endpoint that hosts compute and routes requests. Prebuilt containers (opens in a new tab) provide the web server and framework runtime without requiring a Docker build, while custom containers package full control over dependencies and serving logic. The choice trades maintainability for control over versions, preprocessing, and acceleration.

2.1Prebuilt Containers Versus Custom Containers

Prebuilt containers are versioned Docker images maintained by Google for TensorFlow, PyTorch, XGBoost, and scikit-learn. Each image bundles a specific framework version with CPU or GPU dependencies, and the serving container version must match the framework version used for training. A custom container is a Docker image that packages model artifacts, framework dependencies, and a custom prediction server for deployment on Vertex AI Prediction, GKE, or Cloud Run. The custom container controls the full serving path from model loading to request handling to response delivery.

Choose a custom container when the workload needs private dependencies, non-Python serving code, custom tokenization or ensembles, or low-latency servers with framework-specific optimizations. Start from a Deep Learning Containers or NVIDIA NGC base image rather than a minimal base, because missing CUDA, cuDNN, and framework optimizations directly reduce GPU throughput.

2.2Framework Artifact Contracts for Prebuilt Serving

Exported model artifacts are the files the prebuilt container loads at serving time, and artifacts must comply with the container contract even if training happened outside Vertex AI. Artifacts are staged in a Cloud Storage directory referenced as artifact_uri, and the serving container reads that directory when the model is deployed. TensorFlow models must be exported as a SavedModel directory containing saved_model.pb plus variables and assets. Custom exports without Keras or an Estimator must use the serve tag and serving_default signature, or Vertex AI Prediction cannot discover the serving graph.

PyTorch, XGBoost, and scikit-learn use exact filename contracts that the serving container expects at load time. A PyTorch model served by the prebuilt TorchServe image must be packaged as model.mar created with model-name model, including weights and a handler script that defines preprocessing and postprocessing. An XGBoost model is exported as model.bst with save_model or as model.joblib with joblib, while a scikit-learn model is exported as model.joblib. A scikit-learn Pipeline object chains transformers and the final estimator into one object, so raw features flow through each transformer in order to the estimator without separate serving code.

2.3Custom Prediction Routines for Managed Preprocessing Extensions

Custom prediction routines (opens in a new tab) extend a prebuilt container with custom Python code for input and output transformation. The code runs inside the prebuilt serving image, so traffic flows from the Endpoint to the custom preprocessor, then to the framework predict call, then to the custom postprocessor, and then back to the client. Use custom prediction routines when raw requests do not match model tensors, for example when decoding base64 images, tokenizing text, joining Feature Store values, or remapping labels. The routine preserves prebuilt autoscaling, logging, and monitoring while adding only the transformation layer. Choose a full custom container only when a non-supported framework, custom C++ op, custom server, or system package is required.

2.4Registration and Endpoint Deployment Workflow

A Vertex AI deployment separates the versioned model definition from the serving endpoint. The Model resource binds artifact_uri to serving_container_image_uri, while the Endpoint resource provisions serving compute and exposes the online prediction URL. Prediction traffic flows from clients to the Endpoint and then to the selected deployed model replicas. Deployment follows an ordered workflow:

1. Save locally: Export the SavedModel or framework artifact to a local directory such as ./model/.
2. Stage to Cloud Storage: Copy every artifact file to the serving bucket directory that will become artifact_uri.
3. Upload Model: Call aiplatform.Model.upload with display name, artifact_uri, and serving container image URI.
4. Create Endpoint: Call aiplatform.Endpoint.create with a display name to create the logical serving endpoint.
5. Deploy model: Call model.deploy with endpoint, deployed model display name, and machine type such as n1-standard-2.

Deployment is a long-running operation that can take more than five minutes depending on model size and image pull. An Endpoint can host multiple deployed models behind one address and split traffic by percentage, such as 90% to the stable version and 10% to a canary. Keep the prior version deployed until the new version is validated, because undeploying removes the instant rollback target.

2.5Multi-Framework Optimization for Startup, Acceleration, and Scaling

Container and model packaging for multi-framework serving determines how quickly diverse frameworks start and how efficiently they share accelerators. Choose a model format with fast load times such as GGUF, save the inference-ready model during the image build, and avoid pickle checkpoints for security. Use 4-bit quantized models to maximize concurrency unless testing proves quality loss, because quantization reduces GPU memory per request and increases parallelism. Actively manage context length and reserved GPU memory, since a smaller context window lets more queries run in parallel and memory must be tuned as high as possible without causing an out-of-memory error.

CPU versus GPU placement assigns each framework to the accelerator that fits its dominant math pattern. CPU-optimized backends fit lightweight text models, XGBoost trees, and preprocessing, while GPU and TPU backends fit PyTorch deep learning and LLM inference. GKE Inference Gateway prioritizes GPU and TPU instances for intensive tasks, with detailed health checks ensuring traffic flows only to healthy model servers. Cloud Run scales rapidly and to zero for low-GPU and event-driven work and supports NVIDIA L4 GPUs, while GKE with GPUs or TPUs supports TorchServe, vLLM, TensorRT-LLM, and Text Generation Inference with autoscaling on CPU, memory, or custom GPU utilization. Monitoring combines latency, throughput, GPU utilization, accuracy, skew, and drift signals, with feature attributions showing which inputs drive prediction changes.

Model artifact registration and version lineage records every base, fine-tuned, and custom agent model as a distinct reproducible version with its artifacts and history. Model Registry is the central repository for model versions and metadata, while Artifact Registry is the immutable store for underlying files and containers. Together they link each training pipeline run, its source datasets, and its provenance metadata to a servable version. Only models that pass validation are registered for production deployment, so the registry enforces a gate between continuous training and serving.

3.1Model Registry and Artifact Storage Separation

Model registration in Model Registry creates the single source of truth for model versions by tracking each version, its metadata, and its link to underlying artifacts. Registration applies to base foundation models, fine-tuned derivatives, and custom agent models, with each iteration stored as a new version rather than overwriting a prior version. Artifact storage in Artifact Registry preserves the exact binaries needed to reproduce serving, including model files, pretrained weights, and serving container images with versioning and tagging. Cloud Storage holds validated configuration files and validation artifacts for a durable audit trail.

Each Model Registry version points to tagged artifacts plus explicit runtime dependencies, container specification, and input-output schemas. Before promotion, the artifact must be staged in a sandboxed serving environment to verify that the model can be loaded and invoked. This verification prevents failures where training succeeds but serving fails due to missing dependencies or incompatible schemas.

3.2Version Aliases and Lifecycle Promotion

Semantic version aliasing attaches a movable pointer name to an immutable model version. Each registered version gets a fixed ID such as v3 that never changes and preserves artifacts, training metadata, evaluation results, and lineage, while an alias such as dev, staging, production, champion, or challenger can be reassigned between versions. Agent code references the alias rather than the fixed ID, so resolving production returns whichever version currently holds that alias. When staging moves from v4 to v5, the staging endpoint keeps the same URI and only the registry mapping changes.

Lifecycle promotion reassigns aliases as a version gains confidence from development to staging to production. A version remains development during fine-tuning, becomes staging when serving production-like traffic for integration and safety validation, and becomes production only after meeting release criteria. Promotion follows an explicit sequence:

1. Register and evaluate: Upload the new version with data, code, and metrics and assign the dev alias.
2. Promote to staging: Move the staging alias to the candidate and deploy that alias to the staging endpoint.
3. Promote to production: Move the production alias to the validated version and update the production endpoint.
4. Archive prior versions: Retain superseded versions with labels and access controls for audit without risk of accidental promotion.

3.3Endpoint Traffic Routing for Comparison and Rollback

Deployment routing on serving endpoints distributes live agent requests across one or more deployed model versions. An endpoint has a stable URI and owns a traffic-split map, so traffic flows from the agent client to the endpoint URI to the selected deployed model. Deploying multiple versions behind one endpoint enables controlled exposure without creating new URIs. Canary rollout sends most traffic to the stable version such as 95% to v4 and 5% to v5, traffic splitting divides traffic such as 50% to each version for A/B evaluation, and seamless rollback shifts traffic back to 100% on the prior version. Rollback is fast because the prior version remains deployed and registered; the operation re-points the production alias and resets the split, then undeploys the faulty version only after recovery.

3.4Evaluation Metadata and Readiness Gates

Evaluation metadata attached to a registered version is the evidence package that makes a version reproducible and auditable. Each version links to training parameters, evaluation metrics, dataset version with Knowledge Catalog lineage, foundation-model details, and serving container image tag. Traditional ML metadata includes accuracy, precision, recall, F1-score, AUC-ROC, and log loss, while generative metadata includes toxicity, coherence, factual accuracy, grounding, instruction adherence, and verbosity. Automated MLOps pipelines register this record on creation or modification so traceability does not depend on manual entry.

Readiness validation gates whether a candidate can be registered for production. A validation step runs before registration, and final evaluation uses a separate test split never used for training or tuning. Agent Platform model evaluation (opens in a new tab) measures optimizing and satisficing metrics such as minimum accuracy and maximum latency and size, while Experiments compares challenger against champion on the same split. When monitoring detects data drift or output-quality shifts, alerts trigger rollback to the most recent stable version identified through registry lineage.

3.5Governance with Identity, Audit, and Traceability

Identity and Access Management (IAM) for registry governance controls who and what can register, read, and promote models and artifacts. Permissions are defined for users, groups, and service accounts and applied across Model Registry, Artifact Registry, Cloud Storage buckets, BigQuery datasets, Agent Platform endpoints and pipelines, and compute such as GKE and Cloud Run. The principle of least privilege governs every grant, so a training service account receives read access to training data and write access for artifacts but not write access to production endpoints. Central registry projects host governed versions and workload projects consume them through explicit grants.

Audit logging and tagging provide the continuous record of which data and code produced which version. Cloud Audit Logs capture API calls, data-access events, and configuration changes to registry, storage, and serving resources. Datasets carry business metadata, ownership, sensitivity levels, and lineage linking processed data to raw sources. Validation results flow to BigQuery for historical tracking while validation artifacts are archived in Cloud Storage, so evidence remains queryable long after promotion.

Controlled rollout strategies compare model versions on live traffic while limiting risk and preserving a fast path back to stability. Canary deployment exposes a new version to a small share first and increases that share in bounded stages, while A/B testing splits traffic between champion and challenger to compare quality and serving behavior. Gemini Enterprise Agent Platform decouples the endpoint layer from the models behind it, so one stable endpoint URL can route to multiple DeployedModel resources. Promotion, iteration, and rollback decisions then follow from per-version metrics and predefined success thresholds.

4.1Canary Releases with Progressive Traffic Shifts

Canary deployment with progressive rollout exposes a new model version to a bounded share of live traffic while the stable version continues to serve the majority. Request flow moves from the front-end router to both versions according to configured weights, and progressive rollouts shift those weights over time from the stable destination to the canary destination. A progressive canary follows an ordered sequence:

1. Deploy side-by-side: Deploy the new version alongside the stable version without shifting traffic yet.
2. Shift initial share: Shift a small share first, such as 1% to the new version.
3. Validate health and success criteria: Hold at that share while comparing error rate, latency, and prediction metrics.
4. Increase in bounded steps: Increase the canary share only after the current step succeeds, for example by growing the shifted share by 50% every 5 minutes.
5. Complete or roll back: Complete cutover to 100% when all stages pass, or return traffic to the stable version on degradation.

Controlled ramp-up also protects queues and downstream dependencies from sudden spikes. Cloud Tasks (opens in a new tab) gradually ramps dispatch toward maxDispatchesPerSecond when a queue is new or idle, and the same gradual treatment applies when a release increases traffic to a queue group or rebalances traffic across queues. Limiting initial operations and increasing by no more than 50% every 5 minutes lets model servers and handlers scale safely.

4.2Ordered Rollouts Across Clusters and Fleets

Ordered rollouts sequence progressive exposure from low-risk environments to critical production environments. The mesh-modernization-order label controls sequential order with three waves: early for the first wave, default for the second wave, and late for the final wave. When fleet modernization is triggered, each cluster group modernizes sequentially and waits for automated steps on the current group to complete before beginning the next group. Fleet rollout ordering applies to automated organization-wide scheduling and is set as a project-level label on the fleet host project.

An organization with multiple fleets can combine approaches rather than using one approach for all fleets. Complex fleets can be deferred by pinning them to a legacy implementation while dependencies are remediated, test fleets can use customer-triggered modernization first to validate behavior, and remaining compatible fleets can follow automated scheduling. This combination lets a team validate a new serving stack in non-production fleets marked early and hold a critical fleet at late or deferred until confidence is high.

4.3A/B Testing on a Shared Endpoint

A/B testing for model comparison deploys two versions to the same serving endpoint and splits live prediction traffic between them. Traffic flows from the client application to the Endpoint URL, and the Endpoint forwards each request to one deployed variant according to the configured split. Deploying both variants behind the same URL avoids breaking changes and avoids a sudden cutover. Best practice is to deploy models of the same type to one Endpoint, because mixed types are harder to manage and compare.

Because compute is associated with the DeployedModel rather than the Endpoint, each variant carries its own machine type, accelerator, and replica counts. The default machine type is n1-standard-2, minimum replica count must be at least 1, and Vertex AI Inference autoscaling scales each variant on concurrent requests. Isolation therefore comes from per-variant compute, while data integrity comes from consistent logging and monitoring across variants.

4.4Configuration, Monitoring, and Deployment Decisions

Configuration of an A/B test creates the Endpoint once and deploys each Model variant to it with an explicit traffic allocation. Deployment settings for location, container, and compute are fixed at deployment time and cannot be changed in place; to change them the model must be redeployed. The workflow creates the Endpoint in the serving region, resolves the Endpoint ID, and deploys each variant with display name, machine type, replica bounds, and a --traffic-split mapping that sums to 100. A first deployment uses --traffic-split=0=100, while an A/B deployment uses --traffic-split=0=20,OLD_DEPLOYED_MODEL_ID=80.

Analysis turns the controlled rollout into a promotion, iteration, or rollback choice. Model Monitoring can monitor several versions at once, with online prediction logs and endpoint metrics providing evidence for accuracy, latency, throughput, reliability, and resource efficiency. A version is a candidate for promotion only when improvement persists across enough traffic and time and clears thresholds for quality, latency, and cost. Promotion raises the challenger toward 100% and undeploys the loser, iteration holds a small challenger share while the model or compute is fixed, and rollback shifts traffic back to the prior DeployedModel. The endpoint URL stays constant in all three cases.

Inference preprocessing and postprocessing solutions transform raw requests into model-ready inputs and raw outputs into finished responses without introducing training-serving skew. Vertex AI custom serving runs preprocessing, prediction, and postprocessing inside a containerized HTTP server managed by Vertex AI (opens in a new tab). A custom prediction routine extends a prebuilt image with handler code, while a fully custom container replaces the entire server. Streaming architectures extend the same idea by enriching live events with Feature Store values before calling the endpoint.

5.1Custom Routines and Custom Container Servers

Custom prediction routines and fully custom containers both run on Vertex AI endpoints but divide HTTP server ownership differently. A custom prediction routine keeps the prebuilt server for liveness, health, and routing and adds a custom predictor handler for deserialization and formatting. A custom container requires the team to build and operate the entire HTTP server in any language or framework. Custom code sits on the request path between Vertex AI and the model: traffic flows from the endpoint to the container, the handler transforms the payload, the model produces raw outputs, and the handler transforms those outputs back into the client response.

5.2Container Lifecycle of Liveness, Health, and Inference

The serving container contract requires the Docker image to run an HTTP server indefinitely when the container starts. Common implementations include Flask for lightweight Python servers, TensorFlow Serving for TensorFlow models, TorchServe for PyTorch models, and KServe Python Server for a predictor interface that separates load, preprocess, predict, and postprocess. The start command comes from the Dockerfile ENTRYPOINT, CMD, or both, and Vertex AI can override that behavior with containerSpec.command to replace ENTRYPOINT and containerSpec.args to replace CMD. Environment variables from the image, from Vertex AI, and from the Model resource compose at runtime to configure ports, artifact paths, and per-model behavior.

The container moves through an ordered serving lifecycle:

1. Container start: The startup command launches and the HTTP server begins listening in a running state indefinitely.
2. Liveness checks: Vertex AI sends repeated liveness probes, and the server must respond so the platform does not restart the container.
3. Health checks: Only after the server is live, Vertex AI sends health probes to verify that model artifacts and dependencies are ready, moving the container from starting to healthy.
4. Inference requests: Only after the container is healthy, Vertex AI forwards prediction traffic where the handler deserializes, preprocesses, invokes the model, postprocesses, and returns a response.

5.3In-Artifact Transformations for Framework Exports

In-graph and exported artifact transformations embed preprocessing and postprocessing directly into the served artifact so Vertex AI receives raw payloads and returns finished predictions. The artifact flows from training to Cloud Storage to a Model resource to an endpoint, and the serving container executes the embedded logic on each request. This design eliminates training-serving skew because the same transformation code runs in training and serving, and it reduces latency by avoiding a separate preprocessing hop.

TensorFlow packages computation including in-graph ops as saved_model.pb, with optional config/batching_parameters_config in the same Cloud Storage directory to combine individual requests into efficient executions. PyTorch packages weights and a handler script as model.mar, where data flows from client payload to handler preprocessing to forward pass to handler postprocessing. XGBoost exports as model.bst or model.joblib and scikit-learn exports as model.joblib, with a scikit-learn Pipeline chaining transformers to the estimator. To serve XGBoost or scikit-learn, the prebuilt inference container version must match the training version or the Model cannot load.

5.4Streaming Enrichment and Inference with Feature Store

Low-latency feature enrichment combines a real-time event stream with precomputed features without adding lookup delay. Cloud Pub/Sub carries the live transaction stream, Dataflow runs an Apache Beam streaming pipeline, Vertex AI Feature Store supplies low-latency feature values, and Vertex AI endpoints return predictions. Feature data lives in BigQuery for offline training and history, while online serving uses an online store instance with feature views that map entity keys such as user_id to precomputed features. Because the same BigQuery-backed definitions feed both paths, the serving path reuses the same feature logic and reduces skew.

The Apache Beam enrichment transform performs element-wise enrichment by key-value lookup against Feature Store. Data flows from the Pub/Sub source to the enrichment step, the enrichment step calls Feature Store with the entity key, and returned values flow forward joined to the original event. The transform provides a built-in Vertex AI handler, client-side throttling under burst load, and an optional Redis cache for repeated keys. Streaming inference then uses the Beam RunInference transform with VertexAIModelHandlerJSON to call the deployed model. The end-to-end flow runs in strict order: producers publish events to Pub/Sub, Dataflow reads the stream, enrichment joins Feature Store values onto each event, RunInference scores the enriched JSON record, and the pipeline forwards predictions for storage, alerting, or downstream serving.

Key Facts

• Vertex AI Batch Prediction runs asynchronous jobs where data flows from Cloud Storage or BigQuery through prediction containers to a Cloud Storage directory or BigQuery table, with a default batch size of 64.
• GPU-accelerated batch jobs incur about 10 minutes of startup time, so jobs should target at least 20 minutes to keep most cost on inference.
• Vertex AI dedicated endpoints are single-tenant endpoints required for Model Garden vLLM chat completions and must be created via REST with dedicatedEndpointEnabled set to true.
• A Vertex AI Endpoint hosts multiple deployed models behind one address and splits traffic by percentages that must total exactly 100 to enable canary rollout, A/B comparison, and fast rollback.
• Prebuilt serving requires exact artifact contracts: TensorFlow uses saved_model.pb with serve tag and serving_default signature, PyTorch uses model.mar, XGBoost uses model.bst or model.joblib, and scikit-learn uses model.joblib.