Concept drift happens when the real-world relationship between a model's input data and its target outcome changes, making a model that was trained on old data less accurate over time. Vertex AI Model Monitoring automatically catches this by comparing the distribution of recent prediction data against a baseline from training or an earlier reference period. When the statistical difference exceeds a set limit, the service triggers an alert.
Vertex AI Model Monitoring detects concept drift by calculating how much the data distribution has shifted. It uses two main types of checks: drift detection, which watches how production data changes over time, and skew detection, which compares production data directly to the original training data. For concept drift, the service monitors changes in the target variable or in how features relate to predictions. It uses specific statistical formulas, like Jensen-Shannon Divergence for number-based features and L-Infinity for category-based features, to measure the distance between distributions. Administrators set thresholds for these metrics, and alerts fire when the live data crosses those lines.
When an alert is triggered, an automated workflow can fix the problem without manual steps. The monitoring job writes alert details to Cloud Logging. From there, notifications can be sent through Cloud Monitoring to channels like email, Pub/Sub, Slack, or PagerDuty. The core automation pattern uses these alerts to start a Vertex AI Pipeline that retrains the model with newer data. This creates a closed feedback loop where detection automatically leads to remediation, keeping the model accurate as conditions change.
The automated pipeline follows a specific sequence. First, Vertex AI Model Monitoring samples prediction requests and saves them to a dedicated BigQuery table. Next, at scheduled intervals, it calculates drift metrics by comparing this recent data to the baseline. If a threshold is breached, it logs an alert and sends a notification. This notification, often via a Pub/Sub message, can trigger a Cloud Function or a scheduled Vertex AI Pipeline to begin the retraining process. Finally, the new, retrained model is evaluated and deployed to replace the old one, completing the cycle and ensuring the production system stays reliable.
Vertex AI Model Monitoring is used to set up checks for two common problems: data drift and training-serving skew. It works by establishing a statistical baseline from your training data and then continuously comparing live prediction requests against that baseline using specific distance metrics.
Training-serving skew occurs when the data arriving at a live model endpoint has a different statistical distribution than the data the model was trained on. To configure detection, you create a monitoring job that points to your baseline training dataset, stored in Cloud Storage or BigQuery. As prediction requests come into a Vertex AI endpoint, the service logs them and calculates statistical comparisons. If the difference for any monitored feature grows too large, the system triggers an alert so the team can investigate and potentially retrain the model.
Data drift is the gradual change in production input data over time, measured by comparing recent data against data from an earlier production period (like yesterday versus today). Unlike skew detection, drift detection doesn't need the original training data—it looks for changes within the production stream itself. You configure it by selecting which features to watch and setting numerical thresholds for each. When the live data distribution shifts beyond these limits compared to a recent window, the system flags it, warning operators of changing real-world conditions.
Vertex Explainable AI (XAI) adds another layer of monitoring, especially useful for complex data. It works by calculating how much each input feature influences individual predictions, a value called feature attribution. Vertex AI Model Monitoring can track these attribution scores to detect a subtle problem called feature attribution drift. This happens when the model starts relying on different features to make decisions, which can be an early sign of failure even if the raw input data hasn't changed. This method complements the standard statistical checks on the data itself.
For models built and stored in BigQuery ML, you can use built-in SQL functions to monitor for skew and drift. Functions like ML.DESCRIBE_DATA compute baseline statistics. To check for training-serving skew, you use the ML.VALIDATE_DATA_SKEW function, which automatically compares new serving data against the statistics saved when the model was created. To analyze drift between two time periods in production, you use functions like ML.VALIDATE_DATA_DRIFT. These tools let data analysts perform monitoring directly within their SQL workflows.
This process uses Vertex Explainable AI (XAI) to monitor for changes in how a model makes its decisions, not just changes in the input data. It detects if the model's reliance on specific features shifts over time, which is a form of model degradation that raw data statistics might miss.
Feature attributions are scores from Vertex Explainable AI that show each input feature's contribution to a final prediction. Vertex AI Model Monitoring can track these scores to detect two specific issues. The first is training-serving skew for attributions, where a feature's importance score in production differs from its score in the training data. The second is prediction drift (or feature attribution drift), where a feature's importance score changes significantly over time within the production environment itself.
To set this up, you need a tabular model deployed on Vertex AI with Vertex Explainable AI enabled. For custom-trained models, you must configure the ExplanationSpec during deployment; AutoML Tabular models have this enabled automatically. You also need to prepare baseline data. For skew detection, you must provide your training data (or the results of a Vertex AI Batch Explanation job on that data) in Cloud Storage or BigQuery. For drift detection, a training baseline is optional. It is also recommended to provide a data schema file; if you don't, the monitoring job will remain in a pending state until it can infer the schema from the first 1000 prediction requests.
You create a Model Deployment Monitoring Job on your model's online endpoint to start tracking. In the job configuration, you specify whether to monitor for skew, drift, or both, and point to your baseline data source. Once active, Vertex AI begins logging all prediction requests and their explanation scores to a BigQuery table. The monitoring service then runs periodic analysis, comparing the logged attribution scores against the baseline you defined. You can configure Cloud Monitoring alerts to notify you when the calculated deviation exceeds your thresholds.
You review the results through the statistics and charts in Vertex AI Model Monitoring. A significant shift in a key feature's attribution score often means the feature's real-world relationship to the target has changed, which can hurt prediction accuracy. When an anomaly is detected, the service writes a detailed log entry to Cloud Logging (under the log name featurestore_log). This entry contains the feature name, the measured deviation, the threshold, and the monitoring objective. These logs can be forwarded to services like Pub/Sub to trigger automated diagnostic or retraining pipelines.
Concept drift refers to a change in the relationship between the input features and the target variable the model is trying to predict. Data drift refers only to a change in the statistical distribution of the input features themselves, regardless of their relationship to the target.
Use Vertex Explainable AI monitoring when you need to detect subtle model degradation caused by the model shifting its internal reliance on features, a problem known as feature attribution drift. Standard statistical checks on the raw input data distribution might not catch this type of shift.
The pipeline starts when Vertex AI Model Monitoring detects an anomaly and writes an alert to Cloud Logging. This alert can be published to a Pub/Sub topic, which then triggers an event-driven service like a Cloud Function. The function initiates a Vertex AI Pipeline that retrains the model with recent data, evaluates the new model, and deploys it to replace the degraded version.
You need a deployed custom-trained model with Vertex Explainable AI explicitly configured via the ExplanationSpec. You must also provide a baseline dataset, which is your training data (or a batch explanation of it) stored in Cloud Storage or BigQuery, and it is recommended to provide an analysis instance schema file.
Professional Machine Learning Engineer
Prepare and test your skills
Prepare and test your skills
StudyCerts
StudyCerts