Serving models

Deploying models for batch and online inference using appropriate services (e.g., Agent Platform, Model Garden, Cloud Run, and GKE)

Deploying machine learning models means making them available to run predictions on new data. This is done through two main types of inference: online inference for real-time, low-latency requests and batch inference for processing large datasets at once. Google Cloud Platform (GCP) provides several services suited to these different needs. For online inference, Cloud Run is a fully managed serverless platform that automatically scales containerized models, while Google Kubernetes Engine (GKE) offers more control for complex, high-throughput deployments using Kubernetes orchestration. For batch inference, services like Vertex AI's batch prediction jobs handle large-scale, non-real-time processing. The Agent Platform and Model Garden are higher-level services that can simplify accessing and deploying pre-built models for specific tasks.

Packaging and serving models from different frameworks (e.g., PyTorch and XGBoost) using prebuilt and custom containers

Packaging a model involves bundling the trained model file along with its code dependencies so it can run reliably in any environment. This is most commonly done using containers, like Docker. For models built with frameworks like PyTorch, TensorFlow, or XGBoost, you can use pre-built serving containers provided by Vertex AI, which handle standard prediction requests. When a model requires custom preprocessing, postprocessing, or unique dependencies, you build a custom container. This custom container includes your model artifact, a server that defines the prediction interface (often using a framework like FastAPI or the TensorFlow Serving library), and a list of all required Python packages. The container is then pushed to a container registry like Artifact Registry and deployed to a serving platform like Cloud Run or GKE.

Organizing and versioning models in Gemini Enterprise Agent Platform Model Registry

The Gemini Enterprise Agent Platform Model Registry is a central repository for storing, organizing, and versioning machine learning models. It acts as a single source of truth for models throughout their lifecycle. Each time a new model is trained or updated, it is registered with a unique version tag, allowing teams to track changes and roll back to previous versions if needed. The registry stores not just the model artifact but also associated metadata, such as the training dataset used, performance metrics, and the container image for serving. This organization enables governance, auditability, and simplifies the process of promoting a model from testing to production by ensuring the correct version is deployed.

Implementing model rollout strategies (e.g., A/B testing and canary deployments) to compare model versions

A model rollout strategy is a controlled process for deploying a new model version to users while minimizing risk. Two common strategies are A/B testing and canary deployments. In A/B testing, traffic is split between two or more model versions (e.g., the current version A and the new candidate version B) to compare their performance on live data using predefined metrics. This provides a direct, statistically valid comparison. A canary deployment starts by sending a very small percentage of traffic (e.g., 5%) to the new model version while the rest goes to the stable version. If the new version performs well and shows no errors, the traffic share is gradually increased until it fully replaces the old version. These strategies are implemented using traffic splitting features on serving platforms like GKE or Vertex AI Endpoints.

Developing solutions for inference preprocessing and postprocessing

Inference preprocessing and postprocessing are the steps that transform data before it is sent to the model for prediction and after the prediction is returned. Preprocessing converts raw input data into the format the model expects, such as scaling numerical values, encoding categorical features, or converting text to tokens. Postprocessing converts the model's raw output into a useful business result, such as converting a probability score into a class label or formatting the prediction into a specific JSON structure. These steps are critical for production models and are typically embedded within the serving application itself, either inside the custom serving container or as a separate service that sits in front of the model endpoint. This ensures data consistency between training and serving pipelines.

Key Facts

• Online inference requires low-latency services like Cloud Run or GKE, while batch inference handles large datasets at once using services like Vertex AI batch prediction.
• Models are packaged into containers, using pre-built images for standard frameworks or custom containers for models with unique dependencies and processing logic.
• The Gemini Enterprise Agent Platform Model Registry stores model artifacts, versions, and metadata to provide governance and a single source of truth.
• A/B testing splits traffic to compare model versions directly, while canary deployments gradually increase traffic to a new version to monitor its stability.
• Preprocessing transforms raw input for the model, and postprocessing converts the model's output into a business-ready format, both often implemented within the serving container.