Automating model retraining

Determining an appropriate retraining policy

Determining an appropriate retraining policy involves creating a plan for when and how to update a machine learning model with new data. This is necessary because a model's performance can degrade over time as the data it encounters in the real world changes, a problem known as concept drift or data drift. A good policy balances the cost of retraining with the need to maintain model accuracy, and it can be triggered by different conditions. Common triggers include a schedule, such as retraining weekly or monthly, or a performance-based rule, such as retraining when the model's prediction accuracy drops below a defined threshold. The chosen policy directly impacts the automation pipeline, dictating how often it runs and what data it uses.

Deploying models in continuous integration, continuous delivery, and continuous training (CI/CD/CT) pipelines (e.g., Cloud Build)

Deploying models in CI/CD/CT pipelines automates the process of testing, packaging, and releasing new model versions, and it extends the concept to include automated retraining. A continuous integration (CI) pipeline automatically tests new model code whenever a change is made to the source repository. Continuous delivery (CD) automates the steps to deploy a successfully tested model to a staging environment, and continuous training (CT) adds the automated retraining of the model itself using fresh data according to the retraining policy. On Google Cloud, Cloud Build is a key service for orchestrating these pipelines; it executes a series of defined steps, such as running unit tests, containerizing the model with Cloud Buildpacks, and pushing the resulting model artifact to Artifact Registry. The final step in the pipeline is often deploying the new model version to a serving platform like Vertex AI, which can then perform a canary or A/B test before fully replacing the old version.

Key Facts

• An appropriate model retraining policy is a plan that defines the triggers, such as a schedule or a drop in accuracy, for updating a model to combat performance degradation from concept drift.
• Continuous integration (CI) in machine learning automates the testing of new model code, while continuous delivery (CD) automates the deployment of tested models.
• Continuous training (CT) extends CI/CD by automatically retraining models with new data based on a predefined policy.
• Cloud Build is a Google Cloud service used to orchestrate CI/CD/CT pipelines, handling steps from testing to containerization and deployment.