Prepare and test your skills
Prepare and test your skills
An appropriate retraining policy is a controlled rule that decides what data a model relearns from, how prior learning is reused, and when a new version replaces production. The policy combines a data window with a computation mode and constrains both by label latency, feature freshness, and Cloud compute budget. The goal is sustained performance without training on incomplete labels, stale features, or unbounded cost.
Cold-start retraining and warm-start training are two computation modes that differ in initialization, cost, and adaptation behavior. Cold-start retraining trains from new initialization on the full selected window, so each schedule is an independent job that depends only on the curated window. Warm-start training initializes from the previous production checkpoint and continues optimization on new data, so the new model depends on prior model weights, optimizer state, and compatible architecture. The pipeline order is fixed: first the pipeline assembles the window, then it either initializes fresh weights or loads the prior checkpoint, and only after initialization does optimization run.
| Strategy | Initialization | Compute and Time Cost | Adaptation Behavior |
|---|---|---|---|
| Cold-start retraining | New initialization on full window | High, scales with total window size on every run | Relearns long-term and recent patterns; robust to large drift and architecture or hyperparameter changes |
| Warm-start training | Prior production checkpoint as starting point | Low to moderate, scales mainly with incremental data and fine-tuning steps | Adapts quickly to gradual drift; can carry forward bias, stale representations, or degraded weights if drift is large |
Cold-start retraining is chosen when distribution shift is large, when features or labels were corrected historically, or when the team changes model structure, loss, or major hyperparameters. Warm-start training is chosen when drift is incremental, when retraining frequency is high, and when compute budget cannot support full retraining on every schedule. An expert policy often mixes both modes, using frequent warm-start updates for responsiveness and periodic cold-start runs to reset accumulated error.
Sliding windows and expanding windows are two ways to bound the training data used at each retraining time. A sliding window keeps a fixed-length recent interval, so data flows from the ingestion system into the window and older records flow out of the window as time advances. An expanding window keeps a fixed start point and grows the end point forward, so data flows from ingestion into the window and accumulates without forced eviction.
A sliding window prioritizes current behavior and limits compute cost because training volume stays bounded even as history grows. The tradeoff is loss of rare long-term signals, seasonality, and minority-class examples that fall outside the window. An expanding window preserves long-term context and stabilizes performance on cyclical events, but cost grows with every run and old distributions can dominate recent signals unless sampling, weighting, or decay gives higher weight to recent records. Choose a sliding window when drift is fast and budget requires bounded size; choose an expanding window when long-term patterns matter and sufficient compute exists.
Ground-truth label latency and Feature Store freshness determine when a retraining window is actually trainable. Label latency is the delay between a prediction event and the arrival of its verified outcome, so the training dataset depends on the labeling system and cannot include recent events whose labels have not yet arrived. Feature Store freshness is the delay between real-world state and the point-in-time correct feature values available for training, so the training job depends on the Feature Store snapshot that joins features to labels without leakage.
A robust schedule separates event time from availability time by defining the window on labeled, materialized data rather than on raw event arrival. The gated promotion sequence enforces this order:
If label latency is longer than the desired cadence, the policy must either slow the cadence, train on an older fully labeled window while validating on newer proxy signals, or use a two-tier design where frequent warm-start updates use fast weak labels and infrequent cold-start runs use slow verified labels.
Cloud compute budget constraints turn retraining frequency, window size, and training mode into a joint optimization problem. Every schedule consumes data processing for window assembly, accelerator time for training, and validation and storage for candidate evaluation, so total cost depends on how often the schedule fires multiplied by the cost per run. A drift-triggered schedule fires only when monitoring detects drift beyond a threshold, which saves budget during stable periods but depends on reliable detection and labeled validation data. A time-based schedule fires on a fixed cadence, which is predictable for budgeting but can waste runs when no drift exists or react too slowly when drift spikes between runs.
Expert policies combine both triggers by setting a maximum cadence bounded by label latency and budget, a minimum cadence to avoid churn, and drift thresholds that can fire an early run when degradation is detected. Under tight budgets, reduce window size with sliding windows, prefer warm-start updates, and require a larger expected gain to pass validation gating. Under looser budgets or high-risk applications, favor larger or expanding windows and periodic cold-start runs that re-establish a clean baseline.
Exam tip: Use warm-start training for fast, low-cost adaptation to gradual drift, but require periodic cold-start retraining when drift is large or prior weights may carry forward degraded representations.
Champion-challenger gating is the rule that a retrained candidate becomes production only when it measurably beats or ties the registered champion under identical conditions. The pipeline pulls the champion from Vertex AI Model Registry as the baseline, trains and evaluates the challenger on new data, and writes both results to Vertex AI Experiments for comparison. Promotion depends on the gate passing, so the endpoint serving the champion remains unchanged when the challenger fails. This dependency of deployment on evaluation prevents silent regression from automated retraining.
Offline evaluation gates are automated checks that compare candidate and champion models on held-out and backtest datasets before any live traffic is affected. The gate first computes primary business-aligned metrics such as AUC, F1, RMSE, or MAPE on the same time-split test set, then requires the candidate to exceed the champion by a minimum delta with statistical confidence. The gate then checks secondary constraints including calibration, slice performance, fairness indicators, prediction latency, and model size, so aggregate gains do not hide critical failures. Automated offline gates typically enforce peer criteria that must all be satisfied:
A candidate passes to the next stage only after all blocking metrics pass and guardrail metrics stay within tolerance.
Shadow deployment and canary rollout are live validation stages that differ in user impact and in the failures they can detect. Shadow deployment analysis in Vertex AI Endpoints mirrors a copy of production requests to the candidate while the champion continues to serve all user responses. Traffic flows from the live endpoint to both the champion serving path and the shadow candidate, and prediction logs flow from both models to BigQuery or Cloud Logging for side-by-side comparison. Because shadow responses are never returned to users, teams can measure live feature skew, latency under real load, and agreement with the champion without user impact.
Deployment fallback in Vertex AI is the ordered return of serving traffic to the prior champion version when monitors detect regression. The endpoint retains the prior model version as available, shifts traffic first to canary and then to full rollout, and reverses that order on failure so traffic flows from the candidate back to the champion. The fallback sequence runs in a fixed order:
Rollback triggers combine model quality signals such as drop in accuracy, precision, recall, or business KPI beyond tolerance, with serving signals such as elevated error rate, p99 latency breach, or prediction drift. Live validation strategies differ by user impact and by what must be proven before full rollout:
| Strategy | User impact | Choose when |
|---|---|---|
| Offline evaluation | No live traffic | The decision can be made from historical labels and latency estimates |
| Shadow deployment | No user impact; live requests mirrored to candidate | Live feature distribution, skew, and latency must be validated safely |
| Canary deployment | Small share of users served by candidate | Real user feedback and business KPI response must be measured before full rollout |
Shadow analysis catches serving skew and infrastructure issues that offline tests miss, while canary testing catches engagement effects that shadow scoring cannot observe. An expert policy sequences them from lowest to highest blast radius, so only a candidate that survives offline gates reaches shadow, and only a candidate that survives shadow reaches canary.
Retraining triggering mechanisms are policies that decide when an automated machine learning pipeline starts a new training run. Schedule-based triggers start retraining on a fixed calendar interval, such as daily, weekly, or monthly, regardless of current performance. Data flows from the feature store or BigQuery tables to Vertex AI Pipelines on a Cloud Scheduler cron invocation, and the pipeline depends on the availability of a complete training window at that time. This approach is simple, easy to budget, and auditable, so it fits stable cadence and gradual drift but reacts poorly to sudden shifts.
Event-driven triggers start retraining when a system event signals that new data or context is available. Data flows from the source system to an event bus, typically Pub/Sub or Cloud Storage notifications, which then invokes the Vertex AI Pipelines retraining job only after the event arrives. Common events include arrival of a new partitioned batch in BigQuery, accumulation of N new labeled examples, or deployment of a new feature version. This architecture minimizes wasted compute when arrival is irregular and reduces staleness when volume spikes, but it depends on reliable event delivery, idempotent pipeline design, and guardrails against trigger storms.
Vertex AI Model Monitoring continuously compares live serving data against a training baseline to generate metric-based signals. The monitoring job samples prediction requests and responses from the Vertex AI Endpoint, stores analysis in BigQuery, emits metrics to Cloud Monitoring, and publishes alerts to Pub/Sub when thresholds are breached. Three signal groups drive metric-based retraining:
The optimal trigger architecture selects and combines mechanisms by risk, cadence, and sensitivity:
| Trigger Type | Primary Signal | Choose When |
|---|---|---|
| Schedule-based | Elapsed time | Data cadence is regular, drift is slow, and predictability and cost control matter most |
| Event-driven | Data arrival or system event | Data arrives in bursts or batches, and freshness after arrival is business-critical |
| Metric-based | Drift distance or performance drop | System is highly sensitive to distribution change and unnecessary retraining must be avoided |
In practice, production systems layer the triggers with validation gating. A schedule provides a maximum staleness bound, events provide freshness on arrival, and Model Monitoring metrics provide an early override when drift occurs first. First the trigger fires and stages a candidate dataset, then data validation runs before any training starts, and only after validation passes does the pipeline train, evaluate against the champion, and promote only if the challenger exceeds performance and safety gates.
Continue with this study guide: Determining an appropriate retraining policy
Deploying models in continuous integration, continuous delivery, and continuous training pipelines is the automation that moves ML code and retrained models from source commit to safe production serving. Cloud Build orchestrates testing and containerization of pipeline code, Vertex AI Pipelines executes retraining, and Vertex AI Endpoints serves versioned models with monitoring feedback. The full loop flows from trigger to training to evaluation gate to registry to staged rollout, with monitoring signals flowing back to trigger retraining or rollback.
Continuous Delivery for machine learning is the automated path from a newly trained model to a safe production deployment on Vertex AI Endpoints. The path flows from a continuous training trigger to an automated evaluation gate, then from Vertex AI Model Registry to a staged endpoint rollout, with monitoring signals flowing back to trigger retraining or rollback. Cloud Build orchestrates the pipeline steps, while Cloud Deploy and Cloud Monitoring track rollout status and serving health.
Automated validation gates are pipeline checks that block deployment unless a candidate meets baseline thresholds. Continuous training pipelines start from new data arrival, a monitoring signal such as data drift, or a fixed schedule, then run managed jobs such as Vertex AI custom training for hyperparameter tuning and distributed training or automated fine-tuning for foundation models. Training output flows to artifact storage, and parameters, code versions, dataset versions, and results flow to Vertex AI Experiments for tracking. Vertex AI Model Evaluation runs evaluation automatically, including rapid evaluation, custom metrics, human-in-the-loop review, and adversarial testing. The evaluation gate compares the candidate against explicit thresholds before promotion:
Only a model that meets all thresholds passes the gate and becomes eligible for registration and deployment. A complete CD/CT sequence runs in explicit order:
Vertex AI Model Registry is the versioned system of record that stores approved model artifacts and preserves lineage for governance and reproducibility. After a successful training run, artifacts flow from the training job to Cloud Storage buckets or directly to Model Registry, where each version retains its training data, code, parameters, and evaluation results. This lineage lets a pipeline resolve which version is stable, compare versions, and revert to the most recent stable version when production degrades.
Promotion through environments depends on infrastructure as code integrated into the delivery pipeline. Terraform defines resources such as GKE clusters, endpoint configurations, and Cloud Storage buckets, Cluster Toolkit provides curated Terraform blueprints for HPC and AI clusters, and Config Connector manages Google Cloud resources as Kubernetes objects from YAML manifests. The delivery pipeline depends on Cloud Build and Infrastructure Manager to validate changes with Terraform validate and plan, store versioned configurations as artifacts, maintain separate dev, staging, and prod configurations in version control, and promote automatically on code commits.
Safe deployment to Vertex AI Endpoints is a staged rollout that limits blast radius by exposing a new version to a subset of traffic before full promotion. Traffic flows from clients to the endpoint, and the endpoint splits that traffic between the stable version and the canary version so live performance, latency, and error rate can be compared under real load. Autoscaling then adjusts serving capacity in response to traffic patterns, and the pipeline monitors scaling behavior to confirm it responds correctly.
Availability and rollout health are tracked by integrating Cloud Deploy with Cloud Monitoring. Cloud Monitoring collects request rate, error rate, latency, and resource utilization from endpoints, plus infrastructure signals such as CPU, GPU, and TPU usage, memory, network throughput, and disk I/O from Compute Engine, Google Kubernetes Engine (GKE), and accelerators. Deployment status for canary releases and rollbacks flows from Cloud Deploy to Cloud Monitoring dashboards and alerts, so operators see which version serves each traffic share and whether the canary is healthy enough to promote.
Rollback is the automated return of endpoint traffic from a degraded version to the last stable Registry version. For traditional models, Vertex AI Model Monitoring detects training-serving skew and prediction drift by comparing live inputs against a golden dataset and baseline schema built with TensorFlow Data Validation (TFDV), and it fires alerts when custom thresholds are breached. For generative AI, the Gen AI evaluation service continuously monitors output quality and safety, while Cloud Logging and Cloud Monitoring track shifts in quality or harmful content and trigger the same rollback procedure. The two workload paths share the same delivery pattern but use different gates and monitors:
| Workload | Key production metrics | Evaluation and monitoring tools |
|---|---|---|
| Traditional ML | Accuracy, precision, recall, F1-score, AUC-ROC, log loss, prediction drift and skew vs. golden dataset | Vertex AI Model Monitoring, TFDV baseline schema, custom alert thresholds |
| Generative AI | Response quality, safety, instruction adherence, grounding, writing style, verbosity, harmful-output rate | Gen AI evaluation service, Vertex AI Model Evaluation, Cloud Logging and Cloud Monitoring alerts, human-in-the-loop review |
Both paths require threshold-based gates before registration, versioned lineage in Model Registry, staged endpoint rollout, and alert-driven retraining or rollback. Continuous observability supports this loop with latency, traffic, error rate, and saturation, plus Cloud Logging for events, Cloud Trace for distributed latency, Cloud Profiler for bottlenecks, and Error Reporting for exceptions.
Automated pipeline CI/CD and containerization with Cloud Build is the practice of testing ML pipeline code on every source change and packaging each component into versioned containers. Cloud Build imports source from GitHub, Bitbucket, or Artifact Registry, then executes containerized build steps defined in cloudbuild.yaml to produce artifacts such as Docker containers or Python tar files. Each build step runs in its own Docker container, and steps share the build workspace in order, so test results, built images, and compiled pipeline definitions flow from one step to the next.
Cloud Build triggers are automated rules that execute configured steps whenever changes are pushed to the source. A trigger can fire on any change or only when changes match a branch, tag, or file filter, and separate routines can respond to different triggers for development versus main branches. The build can also be started manually through the CLI or API. Build-time substitutions in Cloud Build are environment variables resolved when a build starts, which makes builds traceable and reusable. Substitutions are used to tag and track exactly what source produced each artifact:
$COMMIT_SHA identifies the source commit and is used to tag per-component Docker images for traceability.$REPO_NAME, $BRANCH_NAME, $TAG_NAME, and $REVISION_ID identify which repository and ref triggered the build, which supports branch-specific routines.$PROJECT_ID and $BUILD_ID identify the Google Cloud project and the specific build execution, even for manually submitted builds.The ML source repository for CI/CD contains all code, definitions, tests, and configuration needed to rebuild the pipeline from source. A complete repository keeps pipeline logic, component logic, containers, tests, and deployment settings together so Cloud Build can validate and package them atomically:
cloudbuild.yaml file, test triggers, and pipeline deployment scripts.settings.yaml that hold pipeline input parameters.Keeping Dockerfiles, component.yaml files, tests, and settings.yaml in the same repository ensures that an image rebuild, a specification update, and a pipeline recompilation all reference the same commit.
The automated test, build, and containerization sequence is the ordered Cloud Build workflow that runs after a developer pushes code from a data science environment such as a local machine or Vertex AI Workbench. The sequence enforces a dependency where containers are built only after tests pass, and the pipeline is compiled only after container URLs are updated. Cloud Build typically performs the following build steps in order:
/workspace directory.$COMMIT_SHA.component.yaml file to point to the newly built and tagged images.pipeline.json definition file.pipeline.json file to Artifact Registry for versioned storage and deployment.The ordering matters because later steps depend on earlier outputs: the image build depends on passing tests, the component.yaml update depends on the pushed image URLs in Artifact Registry, and pipeline compilation depends on the updated component specifications. Artifact Registry is the versioned store that holds both execution environments and pipeline definitions for reliable retraining. Storing custom training and serving containers in Artifact Registry isolates dependencies per task, so feature engineering, training, and inference can evolve without creating a single point of failure. Because pipeline.json is uploaded alongside the images it references, a retraining run can retrieve a matched set of code, environments, and workflow definition from commit SHA to image tag to pipeline definition.
Integration testing and trigger controls prevent wasted builds and verify the full pipeline before promotion. The automated pipeline can be configured to skip triggers when only documentation or experimentation notebooks are modified, so exploratory edits do not rebuild containers or rerun training. For end-to-end verification, the pipeline can be executed as a build test where the build submits a pipeline job and uses the wait() method from the Vertex AI SDK to block until the run completes, and only after successful completion is the pipeline deployed to the target environment.
Exam tip: Tag each component image with $COMMIT_SHA, push it to Artifact Registry, then update component.yaml before compiling pipeline.json, so the compiled pipeline pins the exact images built from that commit.
Event-driven and scheduled continuous training triggers are automated signals that start a Vertex AI Pipelines retraining run without manual intervention. Vertex AI Pipelines executes the full retraining workflow, while Cloud Build automates the surrounding CI/CD/CT lifecycle for infrastructure and pipeline code. Scheduled retraining starts a run on a fixed time cadence even when no explicit event has fired. The scheduled run pulls the current versioned dataset from Cloud Storage buckets, launches managed jobs such as Vertex AI Custom Training for hyperparameter tuning and distributed training, and includes foundation-model fine-tuning when required. Scheduled cadence is chosen when data arrives gradually, when seasonality requires regular refresh, or when governance requires a predictable rhythm.
Event-driven retraining on new data arrival starts a run when fresh training data lands or pipeline code changes. Data flows from the source system to Cloud Storage buckets, and the arrival event flows from storage to the pipeline trigger, so the workflow depends on both the data object and the pipeline definition being versioned. Code-commit triggers flow from version control to Cloud Build, which runs Terraform validate and plan, stores configurations as versioned artifacts, and promotes the updated pipeline. Performance-degradation triggers start a run when Vertex AI Model Monitoring or Cloud Monitoring detects skew, drift, or quality loss. Monitoring signals flow from Model Monitoring to Cloud Monitoring alerting and Cloud Logging, and the alert flows to Vertex AI Pipelines to create a feedback loop for automatic retraining. Common production signals include training-serving skew against the golden dataset, prediction drift, drops in accuracy or AUC, generative output degradation, and latency, traffic, error rate, or accelerator saturation.
Every triggered run moves through validation, versioning, and experiment tracking gates that decide whether it can proceed toward deployment. Every successful run versions code, dataset, and parameters and stores artifacts in Cloud Storage or Model Registry, while Vertex AI Experiments logs parameters and results. A triggered CT run moves through an ordered lifecycle:
The three CT trigger patterns differ by signal source and operational intent:
| Trigger pattern | Signal source | Choose when |
|---|---|---|
| Scheduled | Time cadence managed in CI/CD/CT automation | Data changes gradually or retraining must be predictable and auditable |
| New-data arrival | New objects in Cloud Storage or code commits built by Cloud Build | Fresh examples determine accuracy and waiting would leave the model stale |
| Monitoring-alert | Vertex AI Model Monitoring, Cloud Monitoring, and Cloud Logging alerts | Skew, drift, or quality loss indicates the production model no longer matches reality |
A monitoring alert triggers a new retraining run, but the retrained model still must pass evaluation thresholds and Model Registry versioning before deployment.
A complete automation loop connects timing, learning, and safety into one system. Triggering policies decide when Vertex AI Pipelines should fire, data windows with cold-start versus warm-start modes decide what the model learns at what cost under label and budget limits, and CI/CD/CT automation with Cloud Build, Model Registry, evaluation gates, and shadow, canary, and rollback controls decides whether the candidate is reproducible and safe to keep.
cloudbuild.yaml where per-component images tagged with $COMMIT_SHA are pushed to Artifact Registry before component.yaml is updated and pipeline.json is compiled.Cold-start retraining trains from new initialization on the full selected window, while warm-start training initializes from the previous production checkpoint and continues optimization on new data. Cold-start retraining has high cost that scales with total window size but relearns long-term and recent patterns and is robust to large drift and architecture or hyperparameter changes, while warm-start training has low to moderate cost that scales mainly with incremental data and adapts quickly to gradual drift but can carry forward bias, stale representations, or degraded weights. An expert policy often mixes both modes, using frequent warm-start updates for responsiveness and periodic cold-start runs to reset accumulated error.
Choose a sliding data window when drift is fast and budget requires bounded size, and choose an expanding data window when long-term patterns matter and sufficient compute exists. A sliding window keeps a fixed-length recent interval so training volume stays bounded but can lose rare long-term signals, seasonality, and minority-class examples that fall outside the window. An expanding window keeps a fixed start point and grows the end point forward to preserve long-term context and stabilize performance on cyclical events, but cost grows with every run and old distributions can dominate recent signals unless sampling, weighting, or decay gives higher weight to recent records.
Ground-truth label latency is the delay between a prediction event and arrival of its verified outcome, so the training dataset cannot include recent events whose labels have not yet arrived, and Feature Store freshness is the delay between real-world state and the point-in-time correct feature values available for training. A robust schedule therefore defines the window on labeled, materialized data rather than on raw event arrival and builds the sliding or expanding window only from joined, complete records after label maturation and point-in-time feature join. If label latency is longer than the desired cadence, the policy must slow the cadence, train on an older fully labeled window while validating on newer proxy signals, or use frequent warm-start updates on fast weak labels with infrequent cold-start runs on slow verified labels.
Schedule-based triggers start retraining on a fixed calendar interval regardless of current performance, event-driven triggers start retraining when a system event such as a new partitioned batch, N new labeled examples, or a new feature version arrives via Pub/Sub or Cloud Storage notifications, and metric-based triggers use Vertex AI Model Monitoring to fire when data drift, training-serving skew, concept drift, or performance degradation breach thresholds. Schedule-based triggers are simple, predictable, and easy to budget but react poorly to sudden shifts, event-driven triggers minimize wasted compute and reduce staleness but depend on reliable delivery and guardrails against trigger storms, and metric-based triggers avoid unnecessary retraining but depend on reliable detection and labeled validation data. In practice production systems layer them so a schedule provides a maximum staleness bound, events provide freshness on arrival, and Model Monitoring metrics provide an early override when drift occurs first.
Last updated