Continuous Training (CT) triggers automatically start model retraining workflows in response to scheduled times or specific events. On Google Cloud, these triggers initiate Vertex AI Pipelines, which are services that orchestrate the steps of a machine learning workflow. The main tools for creating these triggers are Cloud Scheduler, Pub/Sub, Eventarc, and Cloud Functions.
Cloud Scheduler creates time-based triggers for Vertex AI Pipelines. You define a schedule using cron syntax, which can be set for regular intervals like nightly or weekly. When the scheduled time arrives, Cloud Scheduler either calls the pipeline directly or sends a message to a Pub/Sub topic to start the pipeline. This method is ideal for routine retraining that doesn't depend on new data or performance changes. You can also manage these schedules by pausing, updating, or deleting them through an API.
Pub/Sub and Eventarc handle triggers based on events. Pub/Sub is a messaging service. For example, when new training data arrives in Cloud Storage, an event notification is published to a Pub/Sub topic, which can then trigger a pipeline. Eventarc is a service that routes events from over 90 Google Cloud sources—like Cloud Storage or BigQuery—directly to services like Cloud Run or Cloud Functions. The event flow starts at the source (like a data job finishing) and goes through Eventarc to the trigger, which finally starts the Vertex AI Pipeline.
Cloud Functions runs small pieces of code in response to events, making it useful for triggers that need custom logic. A Cloud Function can check if specific conditions for retraining are met, such as whether a large amount of new data has arrived or if a model's performance has dropped below a set threshold. If the conditions are satisfied, the function uses the Vertex AI Pipeline API to start the retraining job. This serverless approach scales automatically and only incurs costs when the code runs.
Vertex AI Model Monitoring watches deployed models for problems like data drift or performance degradation. It compares live data against a baseline "golden" dataset. When it detects that drift has passed a configured threshold, it can generate an alert. This alert can be set up to automatically trigger a Vertex AI Pipeline for retraining, creating a feedback loop that fixes model issues without manual steps.
The broader CI/CD/CT system on Google Cloud connects several services. Cloud Build automates building and testing code, pulling from sources like GitHub. It produces artifacts like Docker containers, which are stored in Artifact Registry. Cloud Build can also trigger Vertex AI Pipeline execution as part of its process. The pipelines themselves use specific components for tasks like training jobs, uploading models, running batch predictions, and evaluating model quality before a model is promoted for deployment.
Cloud Build is Google Cloud's service for automating continuous integration, delivery, and training (CI/CD/CT) for machine learning. It executes a series of steps defined in a configuration file to test code, build container images, and update pipeline workflows, enabling reliable updates to ML systems.
The CI/CD architecture for ML uses Cloud Build as the central automation engine. It imports source code from a repository like GitHub. It then runs a build configuration file (usually cloudbuild.yaml), which lists a sequence of steps, each running inside an isolated Docker container. You can use pre-made Cloud Build builders for common tasks or create custom ones. The outputs, like Docker images and pipeline files, are stored in Artifact Registry and Cloud Storage.
Cloud Build triggers automatically start a build when an event happens in the source code repository, such as a push to a branch. Triggers can be filtered to run only when specific files change. Within the build configuration, you use variable substitutions (like $COMMIT_SHA or $BRANCH_NAME) to inject dynamic values at build time. This allows you to tag container images uniquely with the commit hash, making it easy to track which code version produced each image.
A typical ML CI/CD workflow, triggered by a code push, follows these steps in Cloud Build. First, it copies the source code, which includes pipeline code, component specs, Dockerfiles, and tests. It then runs unit and integration tests on the components. If tests pass, Cloud Build builds a Docker image for each pipeline component, tagging each with the commit hash. Next, it uploads these new images to Artifact Registry.
After uploading the images, Cloud Build updates the component specification files with the new image URLs from Artifact Registry. It then compiles the complete pipeline into a single file (like pipeline.json) and uploads that to Artifact Registry too. As a final integration test, the build can optionally run the pipeline on Vertex AI Pipelines to generate a new model, and it can even deploy that model to Vertex AI Inference for API serving. The wait() method can be used to make the build step pause until the pipeline run finishes.
While Cloud Build automates the building and validation of the pipeline definition, the automated execution (Continuous Training) is managed by Vertex AI Pipelines. Other services can trigger a pipeline run programmatically using the Vertex AI SDK. For example, a Cloud Run function subscribed to a Pub/Sub topic can be triggered by messages from Cloud Scheduler (for scheduled runs) or alerts from Cloud Logging (based on Vertex AI Model Monitoring). This creates a chain of automation: Cloud Build updates the pipeline, and event-driven services execute it for retraining.
When setting this up, developers can work in environments like local machines or managed Vertex AI Workbench. Cloud Build triggers can be configured to skip builds for non-code changes. As an alternative to Cloud Build, other CI/CD systems like Jenkins can be used. The automated pipeline can also be set to deploy to different environments (development, test) based on different triggers, with production deployment requiring additional automation managed by Vertex AI Pipelines.
Continuous Delivery (CD) for machine learning automates the steps to validate, register, and safely deploy models. Using Vertex AI and Cloud Deploy, these pipelines ensure new models meet quality standards before they handle live traffic, using strategies like canary releases to minimize risk.
Automated model evaluation acts as a quality check, comparing a newly trained "candidate" model against a current "baseline" model. This happens inside an orchestrated pipeline using Vertex AI Experiments for direct comparison and Vertex AI Model Evaluation for testing on datasets. For predictive models, metrics include accuracy, precision, recall, and AUC-ROC. For Generative AI models, the Gen AI evaluation service in Vertex AI assesses metrics like factual accuracy, toxicity, and safety.
To check for data problems, TensorFlow Data Validation (TFDV) compares the schema and distribution of new data against a golden baseline dataset. This helps catch issues like training-serving skew before a model is approved for deployment.
Vertex AI Model Registry is the central hub for storing and managing model versions. Once a candidate model passes all evaluations, the pipeline registers it as a new version in the registry. The registry tracks the model's lineage, showing its connection to the training data and code. If a newly deployed model has issues, operators can use the registry to quickly identify and revert traffic back to the last stable version.
Safe deployment introduces new models gradually to live traffic to reduce risk. This is orchestrated using Cloud Deploy to deploy models to Vertex AI Endpoints, which host models for real-time online inference. A canary release pattern routes a small percentage (e.g., 10%) of incoming requests to the new candidate model, while the majority of traffic (e.g., 90%) continues to the stable baseline model.
Vertex AI Endpoints can host multiple deployed models at once, allowing teams to adjust the traffic split percentages dynamically. For high availability, endpoints should be configured with at least two worker replicas. The endpoint automatically scales the number of replicas between a minimum and maximum limit based on metrics like CPU usage, and you can update these scaling settings on a live DeployedModel without taking it offline.
After deployment, Vertex AI Model Monitoring continuously watches the live endpoint traffic. It samples the incoming data and compares its distribution against the training baseline to detect prediction drift and skew. For Generative AI models, teams also use Cloud Logging and Cloud Monitoring to track metrics like query latency and token throughput.
When monitoring detects that a metric has crossed an alert threshold, it publishes an alert to Cloud Monitoring. This alert can automatically trigger a rollback to the previous stable model or start an automated retraining pipeline in Vertex AI Pipelines. This creates a closed feedback loop where production problems automatically lead to model updates, helping maintain quality over time.
A scheduled trigger, using Cloud Scheduler, starts a training pipeline at a fixed, recurring time (like every night). An event-driven trigger, using services like Pub/Sub or Eventarc, starts a pipeline in response to a specific event, such as new data arriving in Cloud Storage or a model performance alert.
Use Cloud Functions when your trigger condition requires custom logic or evaluation before starting the pipeline. For example, if you need to check whether the volume of new data exceeds a threshold or if multiple conditions must be met, Cloud Functions can run that code. For simple, direct event routing (like "file uploaded"), Pub/Sub or Eventarc may be sufficient without custom code.
Vertex AI Model Registry stores every version of a model along with its metadata and lineage. When a new model is deployed and later shows problems, operators can use the registry to identify the previous stable version and quickly redirect endpoint traffic back to it, enabling a fast and reliable rollback.
A canary release minimizes risk by exposing a new model version to only a small, controlled portion of live traffic initially. This allows teams to monitor the new model's performance and stability in the real production environment before committing to a full rollout. If issues arise, the impact is limited.
Professional Machine Learning Engineer
Prepare and test your skills
Prepare and test your skills
StudyCerts
StudyCerts