Professional Machine Learning Engineer

Determining an appropriate retraining policy

Determining an appropriate retraining policy involves creating a plan for when and how to update a machine learning model with new data. This is necessary because a model's performance can degrade over time as the data it encounters in the real world changes, a problem known as concept drift or data drift. A good policy balances the cost of retraining with the need to maintain model accuracy, and it can be triggered by different conditions. Common triggers include a schedule, such as retraining weekly or monthly, or a performance-based rule, such as retraining when the model's prediction accuracy drops below a defined threshold. The chosen policy directly impacts the automation pipeline, dictating how often it runs and what data it uses.

Deploying models in continuous integration, continuous delivery, and continuous training (CI/CD/CT) pipelines (e.g., Cloud Build)

Deploying models in CI/CD/CT pipelines automates the process of testing, packaging, and releasing new model versions, and it extends the concept to include automated retraining. A continuous integration (CI) pipeline automatically tests new model code whenever a change is made to the source repository. Continuous delivery (CD) automates the steps to deploy a successfully tested model to a staging environment, and continuous training (CT) adds the automated retraining of the model itself using fresh data according to the retraining policy. On Google Cloud, Cloud Build is a key service for orchestrating these pipelines; it executes a series of defined steps, such as running unit tests, containerizing the model with Cloud Buildpacks, and pushing the resulting model artifact to Artifact Registry. The final step in the pipeline is often deploying the new model version to a serving platform like Vertex AI, which can then perform a canary or A/B test before fully replacing the old version.

Key Facts

• An appropriate model retraining policy is a plan that defines the triggers, such as a schedule or a drop in accuracy, for updating a model to combat performance degradation from concept drift.
• Continuous integration (CI) in machine learning automates the testing of new model code, while continuous delivery (CD) automates the deployment of tested models.
• Continuous training (CT) extends CI/CD by automatically retraining models with new data based on a predefined policy.
• Cloud Build is a Google Cloud service used to orchestrate CI/CD/CT pipelines, handling steps from testing to containerization and deployment.

Frequently Asked Questions

What is the difference between continuous delivery and continuous training?

Continuous delivery (CD) automates the process of deploying a tested model artifact to a serving environment. Continuous training (CT) is a specific extension for machine learning that automates the retraining of the model itself with new data, creating a new artifact that then goes through the CD pipeline.

When should I use a schedule-based retraining policy versus a performance-based one?

Use a schedule-based policy, like weekly retraining, when data patterns change predictably and you want consistent, predictable costs. Use a performance-based policy, triggered by an accuracy drop, when you want to retrain only when necessary to save resources, but this requires robust monitoring to detect the performance decline.

How does Cloud Build fit into a machine learning CI/CD/CT pipeline?

Cloud Build acts as the automation engine for a CI/CD/CT pipeline. It reads a configuration file that defines the steps to test code, build a model container, store it in Artifact Registry, and deploy it to a service like Vertex AI, executing these steps automatically in response to a code change or a retraining trigger.