Prepare and test your skills
Prepare and test your skills
A custom container solution for specialized model serving is a Docker image that packages model artifacts, framework dependencies, and a custom prediction server for deployment on Google Cloud. The image runs on Vertex AI Prediction, Google Kubernetes Engine (GKE), or Cloud Run (opens in a new tab) when prebuilt containers cannot support the required framework, version, preprocessing, batching, or GPU performance. The container controls the full serving path from model loading to request handling to response delivery.
Custom container requirements arise when serving needs fall outside the fixed artifact formats and runtimes of prebuilt containers. Vertex AI Prediction provides prebuilt containers for TensorFlow, PyTorch, XGBoost, and scikit-learn, which require specific exports such as a TensorFlow SavedModel directory with a serve tag and serving_default signature. A custom container does not need to follow those prebuilt contracts, although the same conventions can guide input, output, and signature design.
The choice between container types is a tradeoff between control and setup effort:
| Specification | Prebuilt containers | Custom containers |
|---|---|---|
| ML framework and version | Fixed to one framework and version per image | Any framework, minor version, nightly build, or no framework |
| Dependencies | Common framework dependencies pre-installed | All dependencies pre-installed in the Dockerfile, including private libraries and system packages |
| Prediction server | Managed server with fixed request handling | Custom server such as TorchServe or NVIDIA Triton Inference Server with custom preprocessing, batching, and GPU logic |
| Recommended when | Standard Python model matches an available image | Specialized serving, custom code, non-Python servers, or unsupported framework versions are needed |
Choose a custom container when the workload needs greater customization, private dependencies, non-Python serving code, or low-latency, high-throughput servers with framework-specific optimizations.
Dockerfile design for model serving starts from a validated base image and adds the serving stack in layers. Start from an image in Deep Learning Containers (DLC) or the NVIDIA container registry (NGC) for the framework in use, because those images include current performance-related packages. Do not build a minimal base from scratch, since missing CUDA, cuDNN, and framework optimizations directly reduce GPU throughput and increase maintenance work.
The Dockerfile then pre-installs the application and decides where model weights live. Dependencies, inference code, and the prediction server binary are baked into the image and pushed to Artifact Registry, which provides identity access management (IAM) access control, vulnerability scanning, and observability. Model artifacts can either be baked into the image or loaded at runtime from Cloud Storage (opens in a new tab). Baking the model into the image lets loading benefit from Cloud Run optimized container streaming, but each unique model layer hash creates one stored copy per image and can increase Artifact Registry cost.
Image optimization for serving reduces GPU memory use, increases concurrency, and shortens cold start. Use 4-bit quantized models to maximize concurrency unless quantization is proven to harm result quality, because quantization produces smaller and faster models that need less GPU memory and allow more parallel requests. Ideally models are trained at the target bit depth rather than quantized down after training.
Building large-model images is resource-intensive and often bottlenecked on network throughput. When using Cloud Build (opens in a new tab), build with a high-performance configuration using machineType: 'E2_HIGHCPU_32' and diskSizeGb: '500' to provide increased compute, networking, and disk for the Docker build and push steps. For smaller models, loading the model directly in the container further reduces inference latency by avoiding a separate download at request time.
Model loading performance depends on where the model lives and how traffic reaches it. Models served from Cloud Storage through Cloud Storage FUSE (opens in a new tab) should use parallel downloads with the cache directory as a prefetch buffer, so parts of the model are fetched in parallel from Cloud Storage. For an additional cost, Rapid Cache reduces loading latency by caching data on solid-state drives (SSDs) for faster reads.
Two Cloud Storage FUSE mount options reduce read time for serving pods on GKE and Cloud Run:
cache-dir to cr-volume:{volume name}, for example cr-volume:in-memory-1, and tune additional file-cache flags for cache behavior.enable-buffered-read to true for asynchronous prefetching of Cloud Storage object parts into an in-memory buffer, and tune read-global-max-blocks to control the maximum buffered blocks across all file handles.all-traffic plus Private Google Access, so serving traffic stays on the VPC network.Exam tip: When both options are set, cache-dir takes precedence over enable-buffered-read, and enabling either option counts the Cloud Storage FUSE process under container memory limits, so raise the container memory limit accordingly.
Models loaded from the internet need a high-bandwidth egress path. Route all traffic through the VPC network with Direct VPC egress set to all-traffic and use Cloud NAT to reach the public internet at high bandwidth.
A custom prediction server is the HTTP process inside the container that loads weights, validates request schemas, runs framework inference, and returns predictions. The server implements the specialized logic that prebuilt servers lack, such as custom tokenization, feature transforms, dynamic batching, multi-model ensembles, and GPU execution with TorchServe or NVIDIA Triton Inference Server patterns. The serving deployment depends on the container image in Artifact Registry, the model source in Cloud Storage or the image, and the compute platform in GKE or Cloud Run.
A production serving workload moves data through an ordered sequence:
This architecture uses Cloud Storage as the central, replicated source of truth for training data, checkpoints, and model artifacts, with Virtual Private Cloud (VPC) and Cloud DNS providing networking and discovery. With planning, Cloud Storage FUSE architectures can exceed 1 TB/s throughput with simplified framework access, faster startup without full dataset downloads, and Cloud Storage cost efficiency. Cloud Storage FUSE is not suitable for latency-sensitive workloads with files smaller than 50 MB or that require less than 1 millisecond latency for random input/output (I/O) and metadata access.
Serving infrastructure for multi-framework environments is the shared set of containers, accelerators, load balancers, protocols, and monitors that lets PyTorch, XGBoost, TensorFlow, vLLM, and Triton models run side by side in production. Optimization means matching each framework to the right container build, the right compute, the right scaling policy, and the right protocol, so latency, throughput, cost, and accuracy stay balanced as traffic changes. On Google Cloud, that balance is implemented with prebuilt framework containers, GKE and Cloud Run (opens in a new tab) as serving platforms, model-aware routing, and continuous monitoring for resources and drift.
Container and model packaging for multi-framework serving determines how quickly diverse frameworks start and how efficiently they use memory at inference time. Start from the performance-optimized container image for the framework in use, because those images already include the latest performance libraries, and avoid building a fully custom image unless the framework requires it. Choose a model format with fast load times such as GGUF, which reflects the target quantization type and needs fewer transformations when loaded onto the graphics processing unit (GPU), and do not use pickle-format checkpoints for security reasons. Save the inference-ready model during the image build rather than quantizing or transforming a stored checkpoint at container startup, which shortens cold starts for large PyTorch and large language model (LLM)-based servers.
Quantization and cache warming further increase concurrency across frameworks sharing GPU capacity. Choose 4-bit quantized models to maximize concurrency unless testing proves quality loss, because quantization produces smaller and faster models that reduce GPU memory per request and increase runtime parallelism, and ideally train at the target bit depth rather than quantizing down to it. Actively manage supported context length, since a smaller context window lets more queries run in parallel, and consider a quantized key-value cache when the framework supports it to lower per-query memory and allow higher parallelism at some quality risk. Tune reserved GPU memory for model weights, activations, and key-value caches as high as possible without causing an out-of-memory error.
Build-time cache generation follows an ordered sequence that prepares the container to serve common prompts immediately:
Supply the same cache flags at runtime that were used at build time, and load from the saved model artifact rather than re-applying transforms on startup.
Central processing unit (CPU) versus GPU placement is the decision about which accelerator each framework needs for its dominant math pattern. CPU-optimized backends fit lightweight text models, XGBoost trees, and preprocessing, while GPU and tensor processing unit (TPU) backends fit compute-intensive PyTorch deep learning and generative artificial intelligence (AI) and LLM inference. GKE Inference Gateway prioritizes instances with GPU and TPU accelerators for those intensive tasks, and detailed health checks assess model status so traffic only flows from the gateway to healthy model servers. Use serving frameworks such as vLLM or Triton to expose LLM metrics for scheduling decisions.
GKE custom compute classes control how inference workloads obtain scarce accelerators and how they degrade when capacity is constrained. A custom compute class defines specific compute profiles with fallback policies for autoscaling, for example a profile that gives higher priority to reserved GPU or TPU instances and then falls back to cost-efficient Spot virtual machines (VMs) when reserved capacity is unavailable. For cost and management efficiency across platforms, use a centralized load balancer that routes requests with low GPU needs to Cloud Run and routes more complex, GPU-intensive tasks to GKE, with inter-service policy and communication managed by Cloud Service Mesh. For diverse request mixes, configure content-based routing to direct requests to specialized backends based on URL paths and headers, such as directing image or video model requests to GPU-enabled backends and text-model requests to CPU-optimized backends.
| Platform | Best fit for multi-framework mix | Scaling and accelerator behavior |
|---|---|---|
| Cloud Run | Low-GPU inference, smaller jobs, event-driven preprocessing, and scale-to-zero workloads | Scales rapidly and to zero; supports NVIDIA L4 GPUs for real-time inference with fast cold starts |
| GKE with GPUs | Complex pipelines, TorchServe, vLLM, TensorRT-LLM, and Hugging Face Text Generation Inference (TGI) needing fine-grained orchestration | Autoscales on CPU, memory, or custom GPU utilization; supports rolling updates and YAML-manifest reproducibility |
| GKE with TPUs | Open-model serving with Optimum TPU and vLLM TPU where TPU throughput is preferred | Autoscales TPU pools; isolates TPU-intensive LLM traffic behind model-aware routing |
Exam tip: Set Cloud Run parallelism to less than the GPU quota without zonal redundancy for the project and region, and use a task timeout of one hour or less; deployments fail when parallelism exceeds that quota limit.
Autoscaling and load balancing across frameworks keep heterogeneous containers responsive without paying for idle accelerators. Autoscaling adjusts replica count to current demand, which avoids paying for idle resources while preserving performance at peak load and improving agility and availability. For inference, deploy on scalable platforms such as Gemini Enterprise Agent Platform Inference, GPUs on GKE, or TPUs on GKE, and scale replicas on request rate, latency, and resource utilization, with load balancing distributing traffic evenly across replicas. On GKE, leverage automated scaling on CPU and memory utilization or custom metrics like GPU utilization, plus GKE service discovery for communication between microservices and Pub/Sub for asynchronous patterns.
Traffic flows from clients through load-balancing layers to the framework container that can serve it most efficiently. For global users needing low latency, traffic flows from the client to a global external Application Load Balancer, which routes by geolocation to the closest region with regional endpoint replication and failover, while Cloud Content Delivery Network (CDN) serves static assets and Cloud Monitoring tracks global traffic and latency. For fluctuating AI demand, traffic flows from the load balancer to the GKE Inference Gateway, which performs model-aware routing to the instance whose accelerator and health status best match the request, and serving-framework metrics flow from vLLM or Triton into Cloud Monitoring (opens in a new tab) through Google Cloud Managed Service for Prometheus. Package each pipeline stage — data ingestion, preprocessing, training or fine-tuning, evaluation, and serving — as an individual Docker microservice so each framework scales independently and failures stay contained.
Serving protocols for multi-framework backends define how prediction bytes move between clients, gateways, and backends. Short, stateless predictions work well over REST, while long-running generative AI requests and batch workloads need WebSockets or Google remote procedure call (gRPC) with explicit handling for timeouts and buffering. Configure request timeouts and retries and implement rate limiting and quotas by using API Gateway or Apigee, so one slow LLM stream or large batch does not exhaust shared gateway or container resources. For event-driven preprocessing or smaller inference jobs, trigger Cloud Run functions on events such as a new file in Cloud Storage or a model update in Artifact Registry, and orchestrate multi-step sequences with Workflows or Application Integration.
Semantic and prompt caching at the system-design level reduces repeated work before requests ever reach the GPU. Add semantic caches where appropriate, because caching whole queries and responses limits the cost of common queries, and control variance in preambles because prompt caches are prefix-cached and only useful when prompts arrive in sequence. Insertions or edits in the sequence mean cached prefixes are missing or only partially present, so normalize system prompts and templates across frameworks. Ensure consistent logging and monitoring with Cloud Logging and Cloud Monitoring across Cloud Run, GKE, and mesh-connected services so protocol errors, timeouts, and quota rejections are visible in one place.
Monitoring for multi-framework serving tracks three signals together: model performance, resource utilization, and prediction accuracy drift. Collect AI-specific metrics such as latency, throughput, and GPU utilization, plus model performance degradation metrics such as model drift and data drift, using Cloud Monitoring and Model Monitoring on Gemini Enterprise Agent Platform. Integrate vLLM or Triton metrics into Cloud Monitoring by using Google Cloud Managed Service for Prometheus, configure detailed health checks to assess model status, and export Model Monitoring results to BigQuery for further analysis and tracking across online and batch predictions.
Proactive alerting and reliability practices turn those metrics into action across deployed containers:
Health checks can trigger replacement of faulty nodes, deployment of more capacity, or automatic retraining or fine-tuning with updated data, which prevents both accuracy-based degradation and system-level degradation and improves overall reliability.
Vertex AI Prediction is the managed serving system for online inference on Google Cloud. Model artifacts flow from Cloud Storage to a Vertex AI Model Registry Model resource, and from the Model to a Vertex AI Endpoint that hosts compute and routes prediction requests. Prebuilt serving containers provide the web server, framework runtime, and request handling, so deployment does not require building a Docker image when the model matches a supported framework and version.
Prebuilt serving containers are versioned Docker images maintained by Google for TensorFlow, PyTorch, XGBoost, and scikit-learn. Each image bundles a specific framework version with CPU or GPU dependencies, and the serving container version must match the framework version used for training. Images are published per region, for example us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-13:latest, with europe-docker and asia-docker equivalents, plus tf2-gpu variants for GPU acceleration with CUDA 12.x.
The supported serving stacks divide by framework need:
Choose a prebuilt container when the model uses only standard framework ops and standard request and response tensors. The tradeoff is maintainability versus control: prebuilt images remove patching, serving, batching, and scaling work, but they require the exact artifact layout the image expects.
Exported model artifacts are the files the prebuilt container loads at serving time, and the artifacts must comply with the container contract even if training happened outside Vertex AI. Artifacts are staged in a Cloud Storage directory referenced as artifact_uri, such as gs://BUCKET/DIRECTORY, and the serving container reads that directory when the model is deployed.
TensorFlow models must be exported as a TensorFlow SavedModel directory containing saved_model.pb plus variables and assets. The export path depends on the training API: use tf.keras.Model.save for Keras, use tf.estimator.Estimator.export_saved_model for Estimators, and use tf.saved_model.save or SavedModelBuilder for custom code. Custom exports that do not use Keras or an Estimator must use the serve tag and serving_default signature, or Vertex AI Prediction cannot discover the serving graph.
Two TensorFlow serving extensions build on the same directory layout. For server-side request batching, include config/batching_parameters_config in the same Cloud Storage directory as saved_model.pb following TensorFlow Serving batching configuration. For Vertex Explainable AI, the SavedModel must meet additional export requirements for explanations. If a fully custom container is used instead, this layout can be treated as a guideline rather than a requirement.
Exam tip: A TensorFlow custom export without Keras or an Estimator fails to serve unless it explicitly sets the serve tag and serving_default signature.
Custom prediction routines (CPR) extend a prebuilt container with custom Python code for input transformation and output transformation. The CPR code runs inside the prebuilt serving image, so the request flow stays managed: traffic flows from the Endpoint to the custom preprocessor, then to the framework predict call, then to the custom postprocessor, and then back to the client as the prediction response.
Use CPR when raw requests do not match model tensors, for example when decoding base64 images, tokenizing text, applying feature-store joins, remapping labels, or filtering scores. CPR preserves the prebuilt server, autoscaling, logging, and monitoring while adding only the transformation layer. Choose a full custom container only when CPR is insufficient, such as when a non-supported framework, custom C++ op, custom server, or system package is required.
A Vertex AI deployment separates the versioned model definition from the serving endpoint. The Model resource in Model Registry binds artifact_uri to serving_container_image_uri, while the Endpoint resource provisions serving compute and exposes the online prediction URL. The serving compute depends on the deployed Model, and prediction traffic flows from clients to the Endpoint and then to the selected deployed model replicas.
Deployment follows an ordered workflow:
./model/.artifact_uri.aiplatform.Model.upload with display_name, artifact_uri, and serving_container_image_uri, for example us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-13:latest.aiplatform.Endpoint.create with a display_name to create the logical serving endpoint.model.deploy with endpoint, deployed_model_display_name, and machine_type, for example n1-standard-2.Deployment is a long-running operation that can take more than five minutes depending on model size and image pull. After deployment, the Endpoint serves online predictions from the staged artifacts using the selected machine type and accelerator, and the platform manages replica health, scaling, and request routing.
A Vertex AI Endpoint manages deployment versions by hosting one or more deployed models behind a single serving address. Each deployment creates a distinct deployed model with its own deployed_model_display_name, machine type, and artifact version, while the parent Model resources preserve the registry history. New versions are added by uploading a new Model and deploying it to the same existing Endpoint rather than creating a new Endpoint.
| Resource | Primary unit | Controls |
|---|---|---|
| Vertex AI Model Registry Model | Versioned artifact plus container image | What code and weights are servable, including framework version and CPR logic |
| Vertex AI Endpoint | Set of deployed models plus traffic split | Which version receives live traffic and on what machine type and replica count |
Traffic splitting assigns a percentage of prediction requests to each deployed model on the Endpoint, so 90% can flow to the stable version while 10% flows to a canary. This pattern supports safe rollout, A/B comparison, and fast rollback: shift the split forward only after metrics pass, and shift it back to the prior deployed model if errors or skew appear. Keep the prior version deployed until the new version is validated, because undeploying removes the instant rollback target.
Use a custom container when serving needs fall outside the fixed artifact formats and runtimes of prebuilt containers, such as specialized preprocessing, batching, GPU performance, private dependencies, non-Python serving code, or unsupported framework versions. Prebuilt containers for TensorFlow, PyTorch, XGBoost, and scikit-learn remove patching and scaling work but require specific exports and fixed framework versions, while custom containers package any framework, dependencies, and a custom server such as TorchServe or NVIDIA Triton Inference Server for deployment on Vertex AI Prediction, Google Kubernetes Engine (GKE), or Cloud Run.
A TensorFlow SavedModel must be a directory containing saved_model.pb plus variables and assets, staged in a Cloud Storage directory referenced as artifact_uri. Custom exports that do not use Keras or an Estimator must explicitly set the serve tag and serving_default signature, or Vertex AI Prediction cannot discover the serving graph.
A Vertex AI Endpoint hosts one or more deployed models behind a single serving address and assigns a percentage of prediction requests to each deployed model, such as 90 percent to the stable version and 10 percent to a canary. New versions are added by uploading a new Vertex AI Model Registry Model and deploying it to the same existing Endpoint, and the prior version should stay deployed until the new version is validated to preserve fast rollback.
Last updated