Prepare and test your skills
Prepare and test your skills
Vertex AI Model Monitoring with BigQuery ML compares feature distributions from training baselines to real-time serving payloads to keep production models reliable. A model performs best when serving data resembles training data, and performance can deteriorate when serving data deviates even if the model code has not changed. Monitoring configures baseline datasets and statistical distance measures, such as Chebyshev distance and Jensen-Shannon divergence, to flag when distributions move beyond thresholds and to feed findings back into investigation and retraining.
Training-serving skew is a significant difference between the distribution of feature values in training data and the distribution in production serving data. Data drift is a significant change in feature distribution between spans of production serving data over time, for example between different days of serving data. Skew answers whether production inputs still match what the model learned, while drift answers whether production inputs are moving away from their own history.
| Detection type | Baseline compared from | Serving data compared to | Use when |
|---|---|---|---|
| Training-serving skew | Training statistics saved during model creation | Current serving data payload | Production inputs may no longer match training |
| Data drift | Earlier serving data span | Later serving data span | Production inputs may be changing over time |
Exam tip: Skew compares serving data to training statistics, while drift compares one serving span to another serving span.
Baseline datasets are the reference statistics that later serving payloads are compared against for anomaly detection. For skew detection, training statistics computed during model creation are saved with the BigQuery ML (opens in a new tab) model, so the original training table is not required to run the check. For drift detection, the baseline is a prior serving span, such as a historical table snapshot or features served at a particular point in time retrieved with the ML.FEATURES_AT_TIME function.
The monitoring service depends on matching feature columns to control cost and performance. ML.VALIDATE_DATA_SKEW computes statistics only for serving feature columns that match training feature columns. For long-term comparison, ML.TFDV_DESCRIBE results can be saved to a table with a timestamp, so future serving data can be compared to training and serving data from past points in time.
Basic detection functions in BigQuery ML are SQL functions that compute descriptive statistics and compare datasets to identify anomalous differences. The basic functions handle quick development and monitoring without fine-grained statistics for external integration. The advanced functions produce TensorFlow Data Validation (TFDV) (opens in a new tab) compatible statistics for integration with existing monitoring solutions.
tfdv.generate_statistics_from_csv API.validate_statistics API.These functions move data from source tables to statistics outputs, then from statistics outputs to validation results that flag anomalous features.
Basic skew monitoring follows an ordered workflow from baseline inspection to retraining. The workflow is appropriate when fine-grained statistics are not needed for external integration.
ML.DESCRIBE_DATA on training and serving data to confirm both datasets are within expected parameters.ML.VALIDATE_DATA_SKEW to compare serving statistics to the saved training statistics.Basic drift monitoring follows a parallel workflow that compares serving spans instead of training to serving. The workflow detects change over consecutive spans before retraining becomes too difficult.
ML.DESCRIBE_DATA on training and serving data to confirm expected parameters.ML.VALIDATE_DATA_DRIFT to compare two serving datasets, such as current serving data versus a table snapshot or versus ML.FEATURES_AT_TIME output.Advanced monitoring uses TensorFlow Data Validation statistics for fine-grained control and integration with external monitoring. The advanced path stores statistics as a TensorFlow DatasetFeatureStatisticsList protocol buffer in JSON format, which preserves feature-level detail across time. Statistics flow from source tables through ML.TFDV_DESCRIBE to stored JSON, then from stored JSON through ML.TFDV_VALIDATE to skew or drift findings.
ML.TFDV_DESCRIBE on training and serving tables at intervals appropriate to the monitoring solution and save query results for future comparison.ML.TFDV_VALIDATE with mode SKEW for training-to-serving comparison or drift mode for serving-to-serving comparison, passing the two JSON statistics objects as inputs.A stored serving record typically pairs CURRENT_TIMESTAMP() with the statistics JSON in a table such as serve_stats, so each monitoring run creates a timestamped baseline for the next comparison.
Continuous monitoring jobs in Vertex AI Model Monitoring (opens in a new tab) automatically detect performance degradation, data drift in input distribution, and prediction drift in model outputs. A drift detection job observes prediction logs flowing from the deployed endpoint to the monitoring service, evaluates them against thresholds, and sends results to Cloud Monitoring for dashboards and alerts. Alerts can use notification channels such as email, Slack, or PagerDuty and can trigger automated remediation such as rollback, added capacity, or continuous retraining pipelines.
Visualization and debugging connect monitoring output to operations. Selected BigQuery ML monitoring functions integrate with Vertex AI Model Monitoring charts and graphs for analysis of validation output. Prediction logs are available in Cloud Logging for debugging, validation results can be stored in BigQuery for retrieval and analysis, and comprehensive monitoring output feeds back into continuous training so updated data improves the next model version.
Continuous feature attribution monitoring compares real-time feature importance scores from production against training baselines to detect subtle model degradation. When Vertex Explainable AI (opens in a new tab) is enabled on a deployed tabular model, the monitoring job tracks both input feature distributions and attribution distributions over time. A shift in attribution reveals that the model now relies on features differently than during training, even when raw input metrics look stable.
Vertex Explainable AI attribution scores measure how much each feature contributed to the prediction for a single instance. A prediction request returns only the predicted value, while an explanation request returns the prediction plus per-feature attribution information. Attribution scores are proportional to contribution, are typically signed to show whether a feature pushes the prediction up or down, and must sum across all features to the model prediction score. Because Vertex AI Model Monitoring (opens in a new tab) logs these per-instance explanations from the endpoint, it can track how each feature's contribution distribution changes in production and flag changes that often precede accuracy loss.
Attribution monitoring with explanations enabled detects two distinct attribution failure modes that differ in baseline and time window. Training-serving skew compares production attributions to the training baseline, while prediction drift compares recent production attributions to earlier production windows.
| Mode | Baseline compared against | Production signal |
|---|---|---|
| Training-serving skew | Attribution scores computed from original training data or batch explanation output in Cloud Storage (opens in a new tab) or BigQuery | Production attribution score deviates from training attribution score |
| Prediction drift | Attribution scores from an earlier production time window, with no training data required | Production attribution score changes significantly over time |
Enable skew detection when the training dataset or batch explanation output is available, enable drift detection when it is not, and enable both when continuous coverage is needed.
Exam tip: Training-serving skew requires the training baseline in Cloud Storage or BigQuery, while prediction drift requires no training data and tracks change across production windows.
Prerequisites for attribution monitoring ensure the endpoint emits explanations and the monitoring job can establish a baseline schema. The monitoring job depends on the model deployment, the explanation configuration, and the baseline data source.
ExplanationSpec.ExplanationParameters field is populated, which causes the endpoint to return attributions with predictions.Feature attribution monitoring diagnoses shifts in feature reliance that raw input distribution metrics fail to capture. A model that relied on five features in training and test data but relies on entirely different features in production shows attribution drift even if each individual input distribution appears normal. This early indicator is especially valuable for complex feature types such as embeddings and time series, where traditional skew and drift comparisons are difficult. Because attributions are always numeric regardless of underlying feature type, Model Monitoring can apply standard univariate drift detection to every feature, including multi-dimensional representations.
Attribution analysis starts from the principle that relative feature importance remains relatively stable in a healthy system. A drop in importance for a previously important feature signals that the data source, pipeline, or served population has changed in a way that alters predictions. Interpretation depends on additive and interaction-aware properties that explain how individual contributions combine into the final score.
Common causes of attribution skew and drift include peer failure sources that change what the model sees or what features mean:
Track the most important features first, because the prediction score equals the sum of feature contributions, so large attribution drift in top features usually indicates large drift in model predictions. Monitor all feature representations as single numeric values by summing attributions across embedding dimensions. Account for feature interactions, because attribution to a feature includes both its individual effect and its interactions, so the attribution distribution can change even when the marginal distribution of that feature is unchanged. Monitor feature groups by summing attributions of related features, such as combining grade, sub_grade, and purpose in a credit lending model into a single loan attribution that is tracked as one unit.
Concept drift is a shift in the relationship between inputs and the target over time, where the statistical properties of the target that the model predicts change so predictions become less accurate. Concept drift differs from input change because the model can receive the same features but the correct label for those features is now different. Detecting concept drift requires comparing production predictions against delayed ground-truth labels and then triggering an automated retraining workflow to restore reliability.
Drift and skew monitoring on Google Cloud divides production change by what is compared and what data is required. Model Monitoring on Agent Platform (opens in a new tab) supports training-serving skew detection, prediction drift detection, and feature attribution monitoring for structured numerical and categorical features. Concept drift is the target-distribution case within this family and it cannot be confirmed from inputs alone because it requires ground truth.
| Monitoring type | What is compared | Ground truth required |
|---|---|---|
| Training-serving skew | Training distribution versus live serving input distribution | No, requires pointer to training data in BigQuery, Cloud Storage, or tabular AutoML dataset |
| Data drift | Past production distribution versus current production input distribution | No, monitors enabled features over time when training data is unavailable |
| Concept drift | Production predictions versus delayed ground-truth labels for the target field | Yes, requires joining logged predictions to labels to compute accuracy, precision, recall, F1-score, AUC-ROC, and log loss |
| Feature attribution drift | Baseline feature importance from Vertex Explainable AI versus current attribution distribution | No, uses numeric attribution values as an early proxy for degradation |
Skew detection is preferred when training data is available because deviation of production data from training data is a strong indicator that production performance will not match training performance. Drift detection is used when training data is unavailable or when the goal is to track change over time in production. Attribution drift is especially useful for embeddings and time series where marginal distributions are difficult to compare directly, because attributions are always numeric and additive across dimensions and feature groups.
Exam tip: Data drift and skew need only input distributions, while concept drift needs predictions joined to delayed labels; use output or prediction drift and attribution drift as early signals while waiting for labels.
Concept drift detection joins logged serving predictions to ground-truth labels that arrive later and measures whether classification and ranking quality has degraded. The serving system writes prediction requests to a generated BigQuery table named PROJECT_ID.model_deployment_monitoring_ENDPOINT_ID.serving_predict, and the evaluation job later compares those predictions to the true target column that was excluded from input monitoring. A sustained drop in accuracy, precision, recall, F1-score, AUC-ROC, or increase in log loss against custom thresholds indicates that P(y|X) has shifted, even if input histograms look stable.
Because labels are often delayed by hours or days, production teams use two leading indicators before labels arrive. Output drift tracks unexpected shifts in the prediction distribution or spikes in low-confidence predictions, and attribution drift tracks whether the features driving predictions have changed. If a model relied on five features in training but relies on different features in production, or if attribution to the most important features moves sharply, the prediction score, which equals the sum of feature contributions, will usually drift as well. Common causes include data source changes, schema and logging changes, shifts in user mix or seasonal behavior, and upstream changes from another model that alter coverage or feature meaning.
A stable baseline makes these comparisons trustworthy. Teams construct a golden dataset that represents the ideal data distribution and use TensorFlow Data Validation (TFDV) to analyze training data and establish a baseline schema. Model Monitoring on Agent Platform then compares live input distributions against that golden baseline for skew, compares recent production windows against past production for drift, and visualizes the last 50 monitoring jobs as histograms in the Endpoints console. For batch inference, the same skew metrics are written to Cloud Storage as metrics/skew/feature_skew.json for per-feature statistical distances.
The continuous monitoring architecture connects serving, logging, metrics, and alerting so a distribution shift becomes an actionable signal without manual polling. Prediction traffic flows from the Agent Platform endpoint to the Model Monitoring job, which samples requests at a configured sampling rate and runs on a monitoring frequency with minimum granularity of 1 hour and default of 24 hours. Logged data flows from the endpoint to BigQuery for analysis and to Cloud Logging (opens in a new tab) for centralized collection, while derived metrics flow from Model Monitoring to Cloud Monitoring for dashboards and alert policies.
Alert thresholds control when a statistical distance becomes an incident. Each monitored feature has its own threshold, for example Age=0.4 alerts when the distance between input and baseline for Age exceeds 0.4, and batch monitoring defaults to 0.3 for every categorical and numerical feature when no value is set. Thresholds are tuned by use case, domain expertise, and initial monitoring metrics rather than left at defaults. Model Monitoring on Agent Platform v2 extends this model by monitoring at the model level rather than only the endpoint level and by supporting self-hosted serving on Google Kubernetes Engine (GKE) in addition to managed Agent Platform serving.
Custom observability fills gaps that distribution metrics do not cover. Cloud Logging collects logs from all pipeline components, Cloud Monitoring creates custom dashboards and log-based metrics for error patterns and quality indicators, and notification channels deliver alerts through email, Pub/Sub integrations to Slack, SMS, and PagerDuty, with severity-specific routing. Infrastructure signals such as request rate, error rate, latency, CPU, GPU, and TPU utilization, plus pipeline execution status, flow into the same Monitoring stack so teams can separate model degradation from endpoint outages or pipeline failures.
Automated remediation turns a drift or performance alert into a versioned retraining and redeployment workflow owned by the MLOps pipeline. The feedback loop depends on Model Monitoring for detection, Pub/Sub and Cloud Monitoring for delivery, Agent Platform Pipelines for execution, and Model Registry (opens in a new tab) for lineage and rollback. When the loop is configured, an alert does not merely notify a person; it starts a pipeline that rebuilds the model on fresh data.
The remediation sequence runs in order, with each step depending on the prior step succeeding:
Preparation determines whether this loop runs safely under pressure. Teams identify critical assets such as endpoints, Feature Store instances, datasets in BigQuery and Cloud Storage, and pipeline components, then build playbooks for model rollback from Model Registry, emergency retraining on managed training, and isolation of a compromised data source. Identification and triage use Cloud Logging, Admin Activity and Data Access audit logs, and Security Command Center findings, while containment isolates the endpoint, revokes Identity and Access Management (IAM) permissions, or updates Cloud Armor and firewall rules before eradication removes the root cause in code, data, or upstream models.
ML.VALIDATE_DATA_SKEW detects training-to-serving skew and ML.VALIDATE_DATA_DRIFT detects serving-to-serving drift, while ML.TFDV_DESCRIBE and ML.TFDV_VALIDATE provide fine-grained TensorFlow Data Validation statistics in JSON format.Training-serving skew is a significant difference between the distribution of feature values in training data and the distribution in production serving data, while data drift is a significant change in feature distribution between spans of production serving data over time. Skew detection compares current serving data to training statistics saved during model creation, while drift detection compares an earlier serving span to a later serving span.
ML.VALIDATE_DATA_SKEW computes serving statistics and compares them to training statistics saved at model creation to identify skew between training and serving, while ML.VALIDATE_DATA_DRIFT computes and compares statistics for two serving datasets to identify drift between serving spans over time. ML.TFDV_DESCRIBE and ML.TFDV_VALIDATE provide fine-grained TensorFlow Data Validation (TFDV) compatible statistics in JSON format for integration with external monitoring solutions.
Vertex Explainable AI attribution scores are signed per-feature contributions that sum to the model prediction score, and continuous monitoring compares production attribution distributions against training baselines for skew or earlier production windows for drift. Attribution monitoring catches hidden degradation because the model can rely on different features in production even when individual input distributions appear normal, and attributions remain numeric and additive even for embeddings and time series.
Concept drift is a shift in the relationship between inputs and the target where the same features have a different correct label, so predictions become less accurate even if input histograms look stable. Concept drift detection requires joining logged serving predictions in BigQuery to delayed ground-truth labels to measure accuracy, precision, recall, F1-score, AUC-ROC and log loss, while training-serving skew, data drift and feature attribution drift require only input or attribution distributions and no ground truth.
Last updated