Prepare and test your skills
Prepare and test your skills
Continuous Delivery (CD) for machine learning is the automated path from a newly trained model to a safe production deployment on Vertex AI Endpoints (opens in a new tab). The path flows from a continuous training trigger to an automated evaluation gate, then from Vertex AI Model Registry (opens in a new tab) to a staged endpoint rollout, with monitoring signals flowing back to trigger retraining or rollback. Cloud Build (opens in a new tab) orchestrates the pipeline steps, while Cloud Deploy and Cloud Monitoring track rollout status and serving health.
Automated validation gates are pipeline checks that block deployment unless a candidate model meets baseline performance thresholds. Continuous training pipelines start from a defined trigger, either new data arriving, a monitoring signal such as data drift, or a fixed schedule, and then run managed jobs such as Vertex AI custom training for hyperparameter tuning and distributed training or automated fine-tuning jobs for foundation models. The training output flows from the training job to artifact storage, and parameters, code versions, dataset versions, and results flow from each run to Vertex AI Experiments (opens in a new tab) for tracking.
The evaluation gate compares the candidate against explicit thresholds before promotion:
Vertex AI Model Evaluation runs this evaluation automatically, including rapid evaluation from a provided dataset and prompts, custom use-case metrics, human-in-the-loop review, and adversarial testing with subgroup analysis and counterfactual generation. Only a model that meets all thresholds passes the gate and becomes eligible for registration and deployment, which prevents regressions from reaching production.
Vertex AI Model Registry is the versioned system of record that stores approved model artifacts and preserves model lineage for governance and reproducibility. After a successful training run, artifacts flow from the training job to Cloud Storage (opens in a new tab) buckets or directly to Model Registry, where each version retains its training data, code, parameters, and evaluation results. This lineage lets a pipeline resolve which version is stable, compare versions, and revert to the most recent stable version when production degrades.
Promotion through environments depends on infrastructure as code integrated into the delivery pipeline. Terraform defines resources such as GKE clusters, endpoint configurations, and Cloud Storage buckets, Cluster Toolkit provides curated Terraform blueprints for HPC and AI clusters, and Config Connector manages Google Cloud resources as Kubernetes objects from YAML manifests. The delivery pipeline depends on Cloud Build and Infrastructure Manager to validate changes with Terraform validate and plan, store versioned configurations as artifacts, maintain separate dev, staging, and prod configurations in version control, and promote automatically on code commits.
A complete CD/CT sequence runs in explicit order:
Safe deployment to Vertex AI Endpoints is a staged rollout that limits blast radius by exposing a new model version to a subset of traffic before full promotion. Traffic flows from clients to the endpoint, and the endpoint splits that traffic between the stable model version and the canary version so live performance, latency, and error rate can be compared under real load. Autoscaling then adjusts serving capacity in response to traffic patterns, and the pipeline monitors scaling behavior to confirm it responds correctly.
Availability and rollout health are tracked by integrating Cloud Deploy with Cloud Monitoring. Cloud Monitoring collects request rate, error rate, latency, and resource utilization from endpoints, plus infrastructure signals such as CPU, GPU, and TPU usage, memory, network throughput, and disk I/O from Compute Engine, Google Kubernetes Engine (GKE), and accelerators. Deployment status for canary releases and rollbacks flows from Cloud Deploy to Cloud Monitoring dashboards and alerts, so operators see which version serves each traffic share and whether the canary is healthy enough to promote.
Exam tip: A model must meet its evaluation thresholds to pass the gate and enter Model Registry; traffic splitting on the endpoint only controls exposure after that gate has passed.
Rollback is the automated return of endpoint traffic from a degraded version to the last stable Registry version. For traditional models, Vertex AI Model Monitoring detects training-serving skew and prediction drift by comparing live input distributions against a golden dataset and baseline schema built with TensorFlow Data Validation (TFDV), and it fires alerts when custom thresholds are breached. For generative AI, the Gen AI evaluation service continuously monitors output quality and safety, while Cloud Logging and Cloud Monitoring track shifts in quality or harmful content and trigger the same rollback procedure.
The feedback loop closes when monitoring alerts flow back to orchestration. An alert from Model Monitoring flows to Vertex AI Pipelines to start automatic retraining, and lineage insights from Model Registry identify the stable version that should receive traffic during rollback. Continuous observability supports this loop with the four golden signals of latency, traffic, error rate, and saturation, plus detailed diagnostics from Cloud Logging for events, Cloud Trace for distributed latency, Cloud Profiler for code bottlenecks, and Error Reporting for exceptions.
The two workload paths share the same delivery pattern but use different gates and monitors:
| Workload | Key production metrics | Evaluation and monitoring tools |
|---|---|---|
| Traditional ML | Accuracy, precision, recall, F1-score, AUC-ROC, log loss, prediction drift and skew vs. golden dataset | Vertex AI Model Monitoring, TFDV baseline schema, custom alert thresholds |
| Generative AI | Response quality, safety, instruction adherence, grounding, writing style, verbosity, harmful-output rate | Gen AI evaluation service, Vertex AI Model Evaluation, Cloud Logging and Cloud Monitoring alerts, human-in-the-loop review |
Both paths require threshold-based gates before registration, versioned lineage in Model Registry, staged endpoint rollout, and alert-driven retraining or rollback after deployment.
Automated pipeline CI/CD and containerization with Cloud Build (opens in a new tab) is the practice of testing ML pipeline code on every source change and packaging each component into versioned containers. Cloud Build imports source from GitHub, Bitbucket, or Artifact Registry (opens in a new tab), then executes a series of containerized build steps defined in cloudbuild.yaml to produce artifacts such as Docker containers or Python tar files. This automation connects the data science environment to Artifact Registry and Vertex AI Pipelines (opens in a new tab), so unit tests, integration tests, container builds, and pipeline compilation run the same way for every commit.
Cloud Build triggers are automated rules that execute configured build steps whenever changes are pushed to the build source. A trigger can fire on any change to the source repository or only when changes match certain criteria, such as a branch, tag, or file filter. Separate build routines can respond to different triggers, for example one routine for commits to the development branch and another routine for commits to the main branch. The build process can also be started manually through the CLI or API for one-off runs.
Each build is defined in a build configuration file, typically cloudbuild.yaml, that lists an ordered series of build steps. Each build step runs in its own Docker container, and a step can use a builder image provided by Cloud Build or a custom builder created for ML tooling. Because steps share the build workspace in order, test results, built images, and compiled pipeline definitions flow from one step to the next without manual handoffs.
Build-time substitutions in Cloud Build are environment variables whose values are resolved when a build starts, which makes builds traceable and reusable across branches and commits. Substitutions captured from triggered builds include $COMMIT_SHA, $REPO_NAME, $BRANCH_NAME, $TAG_NAME, and $REVISION_ID, while non-trigger variables include $PROJECT_ID and $BUILD_ID. These variables let the same cloudbuild.yaml behave differently per trigger and let teams reuse a build request with different values.
Substitutions are used to tag and track exactly what source produced each artifact:
$COMMIT_SHA identifies the source commit and is used to tag per-component Docker images for traceability.$REPO_NAME, $BRANCH_NAME, $TAG_NAME, and $REVISION_ID identify which repository and ref triggered the build, which supports branch-specific routines for development versus main.$PROJECT_ID and $BUILD_ID identify the Google Cloud project and the specific build execution, even for manually submitted builds.The ML source repository for CI/CD contains all code, definitions, tests, and configuration needed to rebuild the pipeline from source. A complete repository keeps pipeline logic, component logic, containers, tests, and deployment settings together so Cloud Build can validate and package them atomically. The repository typically includes the following peer artifacts:
cloudbuild.yaml file, test triggers, and pipeline deployment scripts.settings.yaml that hold pipeline input parameters.Keeping Dockerfiles, component.yaml files, tests, and settings.yaml in the same repository ensures that an image rebuild, a specification update, and a pipeline recompilation all reference the same commit.
The automated test, build, and containerization sequence is the ordered Cloud Build workflow that runs after a developer pushes code from a data science environment such as a local machine or Vertex AI Workbench (opens in a new tab). The sequence enforces a dependency where containers are built only after tests pass, and the pipeline is compiled only after container URLs are updated. Cloud Build typically performs the following build steps in order:
/workspace directory.$COMMIT_SHA.component.yaml file to point to the newly built and tagged images.pipeline.json definition file.pipeline.json file to Artifact Registry for versioned storage and deployment.The ordering matters because later steps depend on earlier outputs: the image build depends on passing tests, the component.yaml update depends on the pushed image URLs in Artifact Registry, and pipeline compilation depends on the updated component specifications. A production-ready implementation of this flow is published in the Vertex Pipelines End-to-end Samples on GitHub.
Exam tip: Tag each component image with $COMMIT_SHA, push it to Artifact Registry, then update component.yaml before compiling pipeline.json, so the compiled pipeline pins the exact images built from that commit.
Artifact Registry is the versioned artifact store that holds both execution environments and pipeline definitions for reliable retraining. Artifact Registry receives the per-component Docker images built by Cloud Build and the compiled pipeline.json produced after the image URLs are updated. Storing custom training and serving containers in Artifact Registry isolates dependencies per task, so feature engineering, training, and inference can evolve without creating a single point of failure.
Pipeline compilation translates the Python workflow plus the updated component.yaml files into a portable pipeline.json that Vertex AI Pipelines can execute. Because pipeline.json is uploaded to Artifact Registry alongside the images it references, a retraining run can retrieve a matched set of code, environments, and workflow definition. This linkage from commit SHA to image tag to pipeline definition provides reproducibility across development, testing, and production executions.
Integration testing and trigger controls in Cloud Build prevent wasted builds and verify the full pipeline before promotion to a target environment. The automated pipeline can be configured to skip triggers when only documentation files or experimentation notebooks are modified, so exploratory edits do not rebuild containers or rerun training. Static analysis with Pylint can run as an early quality gate before the more expensive image builds and pipeline runs.
For end-to-end verification, the pipeline can be executed as a build test for integration and regression testing. The build submits a pipeline job and uses the wait() method from the Vertex AI SDK to block until the submitted run completes, and only after successful completion is the pipeline deployed to the target environment. This pattern lets Cloud Build validate data handling, training, evaluation, and serving behavior together, rather than testing components in isolation.
Event-driven and scheduled continuous training (CT) triggers are the automated signals that start a Vertex AI Pipelines (opens in a new tab) retraining run without manual intervention. Vertex AI Pipelines executes the full retraining workflow, while Cloud Build (opens in a new tab) automates the surrounding continuous integration, continuous delivery, and continuous training (CI/CD/CT) lifecycle for infrastructure and pipeline code. The three trigger classes are scheduled cadence, new data arrival, and monitoring signals such as data drift or performance degradation.
Scheduled retraining starts a Vertex AI Pipelines run on a fixed time cadence even when no explicit data or quality event has fired. The scheduled run pulls the current versioned dataset from Cloud Storage (opens in a new tab) buckets, launches managed training jobs such as Vertex AI Custom Training for hyperparameter tuning trials and distributed training, and includes foundation-model fine-tuning jobs when required. Scheduled cadence is chosen when data arrives gradually, when seasonality requires regular refresh, or when governance requires a predictable validation rhythm. Because infrastructure is defined with Terraform and promoted through Cloud Build across dev, staging, and prod, each scheduled run executes against consistent, repeatable infrastructure.
Event-driven retraining on new data arrival starts a Vertex AI Pipelines run when fresh training data lands or pipeline code changes. Data flows from the source system to Cloud Storage buckets, and the arrival event flows from storage to the pipeline trigger, so the workflow depends on both the data object and the pipeline definition being versioned. Code-commit triggers flow from version control to Cloud Build, which runs Terraform validate and plan, stores configurations as versioned artifacts, and promotes the updated pipeline. Choose new-data triggers when model relevance depends on recency, such as changing catalogs, new users, or newly labeled examples, rather than waiting for the next scheduled window.
Performance-degradation triggers start a Vertex AI Pipelines run when Vertex AI Model Monitoring (opens in a new tab) or Cloud Monitoring detects skew, drift, or quality loss in production. Vertex AI Model Monitoring compares live serving input against a golden dataset baseline built with TensorFlow Data Validation (TFDV), and it raises an alert when training-serving skew or prediction drift exceeds custom thresholds. Monitoring signals then flow from Model Monitoring to Cloud Monitoring alerting and Cloud Logging, and the alert flows to Vertex AI Pipelines to create a feedback loop for automatic retraining. For generative AI, the same loop uses the Gen AI evaluation service with Cloud Logging and Cloud Monitoring to track shifts in output quality or harmful content and to trigger rollback or retraining.
The production signals that commonly trigger this closed loop include:
Validation, versioning, and experiment tracking are the gates that decide whether a triggered retraining run can proceed toward deployment. Every successful training run automatically versions code, dataset, and parameters and stores artifacts in Cloud Storage or registers the model in Vertex AI Model Registry (opens in a new tab), while Vertex AI Experiments logs parameters, code versions, dataset versions, and results. Automated evaluation uses Vertex AI Model Evaluation with explicit pass thresholds such as minimum accuracy, maximum error rate, and minimum F1 score, plus generative-specific checks for factual accuracy, safety attributes, and style or format adherence. Only a model that meets all thresholds is considered for deployment, and Model Registry lineage allows revert to the most recent stable version when a later run fails.
A triggered CT run moves through an ordered lifecycle:
The three CT trigger patterns differ by signal source and operational intent. The choice depends on where the signal originates, how quickly the model goes stale, and whether production health is at risk.
| Trigger pattern | Signal source | Choose when |
|---|---|---|
| Scheduled | Time cadence managed in CI/CD/CT automation | Data changes gradually or retraining must be predictable and auditable |
| New-data arrival | New objects in Cloud Storage or code commits built by Cloud Build | Fresh examples determine accuracy and waiting would leave the model stale |
| Monitoring-alert | Vertex AI Model Monitoring, Cloud Monitoring, and Cloud Logging alerts | Skew, drift, or quality loss indicates the production model no longer matches reality |
Exam tip: A monitoring alert triggers a new Vertex AI Pipelines retraining run; the retrained model still must pass evaluation thresholds and Model Registry versioning before deployment.
cloudbuild.yaml where each step runs in its own container, and per-component images tagged with $COMMIT_SHA are pushed to Artifact Registry before component.yaml is updated and pipeline.json is compiled.The ordered sequence is train and log metadata to Vertex AI Experiments, evaluate the candidate against baseline thresholds, register passing models as new immutable versions with full lineage, deploy safely to Vertex AI Endpoints with staged traffic, then observe and react with monitoring signals triggering retraining or rollback. Only a model that meets all evaluation thresholds passes the gate to enter Vertex AI Model Registry and become eligible for deployment.
Choose scheduled retraining when data changes gradually or retraining must be predictable and auditable, new-data arrival retraining when fresh examples in Cloud Storage or code commits determine accuracy and waiting would leave the model stale, and monitoring-alert retraining when Vertex AI Model Monitoring, Cloud Monitoring or Cloud Logging alerts indicate skew, drift or quality loss. A monitoring alert starts a new Vertex AI Pipelines retraining run, but the retrained model must still pass evaluation thresholds and versioning before deployment.
The correct order is copy source into the workspace, run Python unit and integration tests with optional static analysis, then build one Docker image per pipeline component tagged with $COMMIT_SHA and push the images to Artifact Registry. Next update the image URL in each component.yaml file to point to the newly built images, compile the workflow to produce pipeline.json, and upload pipeline.json to Artifact Registry so the compiled pipeline pins the exact images built from that commit.
Last updated