In-graph and exported artifact transformations mean building the feature preprocessing and output postprocessing logic directly into the model file itself. This allows the serving system to accept raw, unprocessed data at inference time. The main benefit is eliminating training-serving skew, because the exact same steps used during model training are automatically repeated when making predictions, since the logic travels with the model.
TensorFlow models must be exported as a TensorFlow SavedModel directory to work with Vertex AI's prebuilt serving containers. Using tf.keras.Model.save will create a SavedModel that can include preprocessing layers as part of the model's graph. The serving signature defined in the SavedModel tells the container how to map raw request data to the model's inputs and outputs, enabling it to handle inference requests directly.
PyTorch models need to be packaged into a model archive file (.mar) that includes a handler script. This script defines how the model processes incoming inference requests. The prebuilt PyTorch serving image expects this archive to be named model.mar. Custom handlers can be written to perform preprocessing transformations inside the serving pipeline, though for very complex needs, a custom container offers more flexibility.
XGBoost models are exported for Vertex AI in one of two specific file formats: model.bst (using xgboost.Booster.save_model) or model.joblib (using the joblib library). The prebuilt container looks for a file with exactly one of these names. Since XGBoost models typically need processed features, any preprocessing like scaling must be handled elsewhere in the inference pipeline if not embedded in the model.
scikit-learn models are exported using the joblib library to create a model.joblib file. A powerful approach is to export a scikit-learn Pipeline object. This Pipeline can chain together preprocessing steps (like a StandardScaler or OneHotEncoder) with the final model, ensuring that the same transformations are automatically applied during inference as during training, which prevents skew.
When prebuilt containers cannot meet specific needs for preprocessing or postprocessing, you can build a custom container. This container must run an HTTP server that responds to liveness, health, and prediction requests (/predict). You can use frameworks like TensorFlow Serving, TorchServe, or a custom Flask/FastAPI server to implement any logic needed before or after the model makes a prediction. This method trades the simplicity of a managed service for complete control over the inference pipeline.
A custom prediction routine is a containerized HTTP server you build and deploy on Vertex AI to serve model predictions when the platform's standard containers are not sufficient. You use this approach to handle unique input formats, apply complex business logic, or perform specialized preprocessing and postprocessing that the prebuilt options don't support.
The custom container image must run an HTTP server that listens for and correctly responds to three types of requests: liveness checks, health checks, and inference requests. The server can be built with any web framework (like Flask or FastAPI) or ML-serving software. It is responsible for loading the model, parsing incoming prediction requests, running the model, applying any custom logic, and formatting the response. Vertex AI sends periodic health probes to the /liveness_check and /health_check endpoints to ensure the server is ready before sending traffic to the /predict endpoint.
Before building a custom container, you should check if a Vertex AI prebuilt container can work. To use one, you must export your model in the framework-specific format it expects: a TensorFlow SavedModel, a PyTorch .mar file, or an XGBoost/scikit-learn .bst or .joblib file. If your serving requirementsâsuch as custom input parsing or complex output formattingâgo beyond what these prebuilt handlers can do, then you must build a custom container.
The core of the custom container is the prediction handler code at the /predict route. This handler must deserialize the incoming request body, which could be JSON, CSV, or another format. It then performs any necessary preprocessing to match what was done during training. After getting a prediction from the model, the handler applies postprocessing logic (like converting scores to labels) and serializes the result for the response. It also handles errors and returns appropriate HTTP status codes.
To deploy a custom container, you build a Docker image with your model, handler code, and dependencies, then push it to a container registry like Artifact Registry. You then create a Vertex AI Model resource pointing to your custom container image. Finally, you deploy this model to a Vertex AI Endpoint for online serving. Vertex AI sets environment variables inside the container, such as AIP_MODEL_DIR which points to your model files in Cloud Storage. The platform manages the container's lifecycle, scaling the deployment automatically based on traffic to the endpoint.
Architecting low-latency feature enrichment and stream transformations involves designing a pipeline that handles real-time data for immediate model inference. In Google Cloud, this typically uses Cloud Pub/Sub for ingesting streaming events, Dataflow for processing them, and Vertex AI Feature Store for quickly looking up precomputed features. Data flows from sources into Pub/Sub, through Dataflow for enrichment and scoring, and finally to services that use the predictions. This design ensures features are consistent between training and inference, preventing skew.
Cloud Pub/Sub is a managed messaging service that decouples data producers from processing systems. Event sources publish messages to Pub/Sub topics. Pub/Sub stores these messages reliably and can apply flow control to manage traffic spikes, protecting downstream systems. It also encrypts messages for security. A Dataflow streaming pipeline then pulls these messages from a subscription connected to the topic.
Dataflow is a serverless service for stream and batch processing, built on Apache Beam. In a streaming inference pipeline, Dataflow consumes raw records from a Cloud Pub/Sub subscription. It performs operations like data cleaning, type conversion, and scaling on each record. Dataflow automatically scales its computing resources up or down based on the volume of messages, maintaining processing throughput without manual intervention.
Vertex AI Feature Store is a managed service for storing and serving machine learning features with very low latency. BigQuery acts as the offline repository for historical feature data. Feature values are synchronized from BigQuery into an online store (backed by services like Cloud Bigtable) for fast access. When an inference request arrives in a streaming pipeline, it uses an entity key (like a user ID) to query the online store and retrieve the latest feature values in milliseconds.
Within a Dataflow pipeline, an Apache Beam enrichment transform can fetch features from Vertex AI Feature Store to add to streaming data before inference. The transform extracts entity IDs from incoming Pub/Sub messages and queries the online store. To improve performance and manage quotas, it can use client-side throttling and an optional Memorystore for Redis cache to avoid repeated lookups. Once the event payload is enriched with features, the pipeline sends it to a RunInference transform or directly to a Vertex AI endpoint for real-time model scoring.
Preventing training-serving skew is achieved by using the same centralized feature definitions and values for both model training and online inference. Vertex AI Feature Store ensures this consistency. During training, point-in-time lookups retrieve only the historical feature values that existed before a given event, preventing data leakage. For production security and performance, pipelines can use private endpoints with VPC Network Peering and VPC Service Controls to keep feature lookups within a trusted internal network.
StudyCerts
StudyCerts/liveness_check, /health_check, and /predict requests, providing full control over the inference pipeline.Prebuilt containers are provided by Vertex AI for major frameworks (TensorFlow, PyTorch, etc.) and require your model to be exported in a specific format. They simplify deployment but offer limited customization. Custom containers are built by you, allowing for any preprocessing, postprocessing, or input/output formatting, but require you to manage the container's HTTP server and dependencies.
Use Vertex AI Feature Store when your model relies on features that are computed or updated separately from the incoming event stream (like a user's average purchase value). It provides a low-latency way to enrich each real-time event with these precomputed features just before inference, ensuring consistency and avoiding complex, real-time calculations in the pipeline.
A scikit-learn Pipeline chains together preprocessing steps (like scaling and encoding) with the final model estimator. When this entire Pipeline object is exported as the model.joblib artifact, the preprocessing logic is saved within it. During inference, the serving code loads the entire Pipeline, which automatically applies the exact same transformations before making a prediction, guaranteeing consistency with training.
Professional Machine Learning Engineer
Prepare and test your skills
Prepare and test your skills