Prepare and test your skills
Prepare and test your skills
Containerized preprocessing for serving is the practice of packaging transformation code, library dependencies, and the trained model together inside the serving container that handles prediction requests. On Vertex AI Inference (opens in a new tab) (formerly Agent Platform Inference), the container receives the raw prediction request, runs the identical preprocessing steps used in training, passes the transformed features to the model, and returns the prediction. Because preprocessing executes inside the same container boundary as inference, training and serving cannot drift apart through separate code paths or mismatched libraries.
Embedded preprocessing in the serving container prevents training-serving skew by forcing one implementation of transformations to run in both phases. When preprocessing lives outside the container in a separate client, gateway, or feature pipeline, small differences in scaling, encoding, imputation, or vocabulary mapping change the feature distribution the model sees in production. A self-contained serving container removes that split by making the request flow from raw input to preprocessed features to model output inside one versioned artifact. The container therefore depends on three coupled parts: the model artifacts, the preprocessing logic, and the pinned dependencies that implement that logic.
Isolating machine learning operations (MLOps) stages into different containers further protects consistency without creating a single point of failure. Feature engineering, training or fine-tuning, and inference each use a container with its own dependency management, so a change to training libraries does not silently alter serving behavior. For consistency across the organization, teams extend the prebuilt container images that Vertex AI provides and maintain verified base images in a central repository managed by a platform team.
Custom prediction routines on Vertex AI are the mechanism for running custom preprocessing and postprocessing together with a model on a prebuilt serving stack. The routine defines a model signature with a predefined input and output schema, so Vertex AI knows how raw prediction requests map to model inputs and how model outputs map to responses. At serving time, traffic flows from the endpoint to the container, then in order through preprocessing code, model inference, and postprocessing code before the response leaves the container. Choose this path when the framework is already supported by a prebuilt container but the model needs identical custom transforms before every prediction.
Custom containers for prediction are user-built container images that package the full serving application, including the web server, preprocessing code, model loader, and all dependencies. A custom container for Vertex AI Prediction or AI Platform Prediction executes the same preprocessing steps used during training before making inferences, with complete control over operating system packages, framework versions, and transformation libraries. The request lifecycle follows an ordered sequence:
This sequence keeps raw data on one side of the container boundary and model-ready features on the other, with no external preprocessing service in between. Use Private Service Connect private endpoints to expose the endpoint inside a Virtual Private Cloud (VPC) network, so model interactions during inference stay off public traffic while the container still enforces its fixed input and output schema.
The choice between serving container types is a decision about how much custom transformation logic must travel with the model:
| Option | Preprocessing control | Dependency control | Choose when |
|---|---|---|---|
| Prebuilt containers | Limited to supported frameworks such as TensorFlow and PyTorch, plus custom prediction routines for pre- and postprocessing | Managed base image from Vertex AI, extended only through allowed customizations | The framework is supported and custom prediction routines can express the identical training transforms |
| Custom containers | Full control to embed any preprocessing, feature, and postprocessing code in the image | Full control to pin operating system packages, framework builds, and private libraries | The model needs libraries, native code, or complex transforms that prebuilt containers cannot reproduce exactly |
Prebuilt containers reduce operational work because Vertex AI manages the serving stack and the team only supplies compliant model artifacts and routine code. Custom containers maximize fidelity because the exact training preprocessing module is copied into the serving image, guaranteeing that serving depends on the same code and versions as training. In both cases, creating explicit model signatures with fixed input and output schemas prevents schema drift between training examples and live requests.
Exam tip: Use a custom prediction routine when a prebuilt container supports the framework but you need identical custom preprocessing; use a full custom container when dependencies or transformation logic cannot be expressed on the prebuilt stack.
Versioning and hardening the serving container preserves preprocessing consistency over time and across environments. The serving image, model files, and preprocessing package are stored as versioned artifacts so a rollback restores both the model and its transforms together. Model Registry tracks model versions and metadata and links each version to its underlying artifacts, while the container image itself is versioned and tagged for controlled rollout.
To build and govern serving images securely:
These controls connect directly to safe release: traffic splitting on Vertex AI endpoints or canary deployment on GKE routes a small subset of production traffic to the new preprocessing-plus-model version, monitoring tracks latency, error rate, and prediction drift, and traffic rolls back to the previous stable version if thresholds are missed.
Implementing transformation artifacts and versioning is the practice of packaging data preprocessing logic into a reusable, versioned object and releasing it together with the model binary so training and serving apply identical transformations. A transformation artifact captures operations such as normalization, bucketing, vocabulary mapping, and encoding, plus the parameters learned from training data. When the artifact version is pinned to a model version, the serving path cannot drift from the training path, which prevents training-serving skew and preserves model reliability.
A reusable transformation artifact is a self-contained preprocessing package that can be executed both during training and during inference. Examples include a TensorFlow Transform graph, a scikit-learn preprocessing pipeline, an Apache Beam Dataflow pipeline, or a custom container that embeds the preprocessing code. The artifact depends on versioned code in Git and versioned data in Cloud Storage (opens in a new tab) or BigQuery, and the training job depends on the artifact to produce transformed features. Dataflow handles Apache Beam batch and streaming transforms, Managed Service for Apache Spark handles Spark transforms, and BigQuery handles SQL-based transforms, while TensorFlow Data Validation (TFDV) validates schemas and detects anomalies before the transform runs.
The artifact must travel with the model through the lifecycle in order. First, the preprocessing component builds the transform from the training dataset version and exports the graph or pipeline object. Then, training consumes only transformed data produced by that exported artifact. Then, evaluation, deployment, and serving reuse the same exported artifact rather than reimplementing the logic. Agent Platform Feature Store strengthens this consistency by centralizing feature definitions with an online store instance and feature views backed by BigQuery, so offline training and low-latency online serving read the same feature logic and historical values.
Artifact Registry (opens in a new tab) is the immutable store for versioned build outputs, including serving and training container images, pipeline components, custom ML packages, and validated configuration files. Artifact Registry provides versioning and tagging, integrates with Cloud Build and Cloud Deploy, and supports vulnerability scanning with Artifact Analysis and deployment gating with Binary Authorization. Model Registry is the system of record for model versions and their metadata, where each version links to its underlying artifacts, training parameters, evaluation metrics from validation pipelines, dataset version, code version, and framework or foundation-model details.
The two registries divide ownership of the release:
Use a clear versioning scheme such as semantic versioning for every model version, and attach comprehensive metadata at registration time. Programmatic registration through the Agent Platform software development kit (SDK) or API lets a Agent Platform Pipelines run automatically register the model, transform artifact, dataset version, and evaluation result as one traceable release, and Cloud Build can build containers, version artifacts, and trigger deployment to Agent Platform endpoints from that same pipeline event.
Consistent preprocessing requires that the training service and the prediction service load the same artifact version. During custom training on Agent Platform Training, the job pulls the pinned transform version from Artifact Registry and the pinned dataset version from Cloud Storage or BigQuery, learns transform parameters, and outputs transformed training data plus the fitted transform. The fitted transform is stored alongside the trained model binary, and Model Registry records the binding between model version, transform artifact version, dataset version, and code version.
At serving time, the endpoint pulls the same bound pair from the registries. Traffic flows from the client request to the Agent Platform endpoint, then through the serving container where raw input first passes through the pinned transform and then through the model binary for prediction. To preserve the contract, define a model signature with a predefined input and output schema and deploy with prebuilt containers when the framework fits or custom containers when preprocessing needs custom code, libraries, or co-hosted dependencies. Isolate feature engineering, training, and inference into separate containers so each stage has only the dependencies it needs and a change in one stage cannot silently alter another.
Exam tip: Artifact Registry stores and versions the files — images, components, packages, and configs with tags — while Model Registry tracks and versions the model release — binaries plus metadata linking to the dataset, code, transform, and metrics.
Versioned transforms are governed by access control, lineage, and continuous validation. Identity and Access Management (IAM) (opens in a new tab) enforces least privilege with dedicated service accounts, for example a training pipeline identity that can read only one training bucket and write only to Model Registry, optionally constrained by IAM Conditions such as allowing pipeline triggers only from a trusted Cloud Build trigger. Stringent access policies apply to both Model Registry and Artifact Registry, and Cloud Audit Logs streamed to Cloud Logging and exported to BigQuery capture API calls, data access, and configuration changes for endpoint creation, model deployment, and artifact updates.
Lineage connects raw data to prediction for debugging and compliance. Agent Platform ML Metadata logs data sources, transformations, parameters, and experiment results; Knowledge Catalog tracks pipeline artifact lineage and enables search across model and dataset resources; and automated registration inside MLOps pipelines ensures every new transform and model version appears in the catalogs upon creation. Validation results and drift signals flow from TFDV checks, BigQuery data-quality scans, and Model Monitoring back into retraining decisions, so a detected shift in input distribution triggers a new versioned transform and model pair rather than an untracked hotfix to serving logic.
Orchestrating consistent feature engineering means defining the complete machine learning (ML) workflow as one automated pipeline where the same preprocessing code produces features for both training and serving. In Google Cloud, Vertex AI Pipelines (opens in a new tab) (managed Kubeflow Pipelines) acts as the central orchestrator, while preprocessing components run on scalable engines and TensorFlow Data Validation (TFDV) generates statistics, enforces schema, and detects anomalies. This design prevents training-serving skew because transformations are versioned, validated at component boundaries, and reused rather than reimplemented for production.
Vertex AI Pipelines is the managed orchestration service that automates every step of the ML lifecycle from data preparation through training, evaluation, and deployment. Pipelines are defined programmatically with the Kubeflow Pipelines (KFP) SDK or TensorFlow Extended (TFX) SDK as modular components with typed inputs and outputs, so data flows explicitly from one component to the next. The pipeline service depends on versioned components, parameters, and configurations to make each run reproducible and auditable. Choose Vertex AI Pipelines when the workflow must be scheduled, triggered, monitored, and reused across projects rather than run manually.
A consistent end-to-end run moves through an ordered sequence:
Recurring work is automated with scheduled pipeline runs or triggers on new data or code changes, and each run uses automated parameterization and versioning to create a version history. Workflow health is observed with execution metrics, built-in logging and tracing, and integration with Cloud Monitoring alerts. Reusable pipeline templates standardize the same preprocessing, training, and evaluation structure across teams.
Embedded preprocessing means the transformation logic lives inside a pipeline component that both training and serving paths depend on, so serving requests receive identical treatment to training examples. The preprocessing component reads from governed sources, applies transformations, and emits processed datasets plus transformation artifacts that downstream training and serving components consume. Because the pipeline owns the transform, a change to feature logic requires a new versioned pipeline run rather than a separate serving-side patch. This ownership boundary is what maintains identical transformations and model reliability.
The choice of execution engine depends on the processing model:
| Option | Engine | Choose when |
|---|---|---|
| Dataflow | Apache Beam for batch and stream processing | The work needs unified batch and streaming transforms with ML-component integration |
| Managed Service for Apache Spark | Apache Spark serverless | The work is Spark-based data processing without managing clusters |
| BigQuery | SQL-based processing | The transform is naturally expressed as SQL and benefits from scalable warehouse queries |
Dataflow can combine with Eventarc for event-driven ingestion, while BigQuery also provides automated data quality scans that complement code-based preprocessing. For tabular features, Vertex AI Feature Store maintains consistency and versioning by storing all feature data, including historical offline data, in a BigQuery table or view. An online store instance with defined feature views then serves low-latency features from that same BigQuery source, so offline training reads and online serving reads resolve to identical feature definitions.
TensorFlow Data Validation (TFDV) is the library for generating feature statistics, inferring schema, and validating data inside ingestion and preprocessing pipelines. TFDV computes distributions over training data, logs those statistics as pipeline artifacts, and enforces the expected schema at component boundaries for raw data, processed data, and serving requests. When live serving data violates the training schema or shows anomalous distributions, the validation step flags the mismatch before errors propagate into predictions. This schema boundary between training expectations and serving reality is the primary control against skew.
Large-scale and programmatic checks extend TFDV within the same pipeline:
Validation results and metrics flow from pipeline components into BigQuery for analysis and historical tracking, while validation artifacts are archived in Cloud Storage (opens in a new tab). That stored history lets teams compare current serving statistics against prior training baselines, debug unexpected behavior, and prove which schema version governed a specific run.
Exam tip: Use TFDV to enforce schema and detect anomalies on pipeline data and serving requests, while BigQuery handles large-scale SQL quality checks and Dataflow handles programmatic streaming or batch validation.
Configuration validation, lineage, and continuous monitoring keep feature engineering consistent after deployment by validating inputs and tracking drift against the training baseline. Pipeline configuration files in YAML or JSON are validated against schemas with libraries such as jsonschema before a run starts, command-line arguments and pipeline parameters are checked for types, valid values, and required fields with argparse, and Vertex AI Pipelines enforces expected types and properties through built-in component input validation. Validated, versioned configuration files are stored in Cloud Storage or Artifact Registry to ensure reproducibility and maintain an audit trail. File-level checks validate format correctness for CSV, Parquet, and images, plus size and checksum validation in Cloud Storage to detect corruption or incomplete transfers.
Once deployed, consistency is monitored continuously against the training baseline. Model Monitoring on Vertex AI detects data drift in input distributions compared to training and prediction drift in model outputs, with drift detection jobs, thresholds, and alerts integrated into Cloud Monitoring dashboards and notification channels. Feature serving is monitored for drift, prediction distributions are checked for shifts, and output schema is validated with alerts for unexpected outputs. Monitoring output feeds back into continuous training, so a confirmed drift or anomaly starts a new pipeline run that regenerates statistics, revalidates schema, retrains on current data, and redeploys through the same versioned path.
Containerized preprocessing for serving prevents training-serving skew by packaging transformation code, library dependencies, and the trained model together inside the same serving container so raw requests flow through preprocessing, inference, and postprocessing in a fixed order. Because one implementation of transformations runs in both training and serving inside one versioned artifact, separate code paths or mismatched libraries cannot change the feature distribution the model sees in production.
Use a custom prediction routine when the framework is already supported by a prebuilt Vertex AI container but the model needs identical custom transforms before every prediction. Use a full custom container when libraries, native code, or complex transforms cannot be reproduced exactly on the prebuilt stack, because the custom container packages the full serving application with complete control over operating system packages, framework versions, and transformation libraries.
Artifact Registry is the immutable store for versioned build outputs such as serving and training container images, pipeline components, custom machine learning packages, and validated configuration files. Model Registry is the system of record for model versions and their metadata, where each version links to its underlying artifacts, training parameters, dataset version, code version, transform artifact version, and evaluation metrics.
Last updated