Monitoring AI solutions

Identifying risks to AI solutions

Sources of Performance Degradation and Data Drift

Model degradation occurs when a deployed machine learning model experiences a decrease in prediction accuracy over time due to changes in real-world environments. Training-serving skew arises when the data used to train a model differs from the live data it encounters during production inference. This discrepancy typically happens because of inconsistent data preprocessing pipelines, differences in feature generation code, or time delays between feature capture and prediction. In contrast, data drift represents a statistical shift in input feature distributions over time, while concept drift occurs when the relationship between input features and the target label changes. Left unmanaged, both drift and skew cause the model to make inaccurate inferences without throwing explicit software runtime errors.

Security, Fairness, and Operational Vulnerabilities

Production AI solutions face operational, ethical, and adversarial risks that threaten system reliability and integrity. Adversarial attacks introduce subtly modified inputs designed to mislead a model into producing incorrect high-confidence predictions. Data poisoning risks occur during retraining cycles when unverified or malicious data enters the training pipeline. Fairness risks involve unintentional demographic bias, where model performance varies across sensitive groups. Teams evaluate feature attributions using Vertex Explainable AI to identify whether models rely on inappropriate feature correlations, utilizing methods such as Shapley values for tabular models or Integrated Gradients for deep neural networks.

Monitoring, testing, and troubleshooting AI solutions

Continuous Monitoring with Vertex AI Model Monitoring

Vertex AI Model Monitoring is a managed service that automatically detects feature skew and feature drift for models deployed to Vertex AI endpoints. For skew detection, the service compares incoming production request payloads against the original baseline training data stored in Cloud Storage or BigQuery. For drift detection, the service samples production traffic over consecutive rolling time windows and compares the current distribution against a baseline of historical serving data. Vertex AI Model Monitoring calculates statistical distance metrics—such as the Jensen-Shannon divergence or L-infinity distance—for each feature and emits an alert through Cloud Monitoring when a metric exceeds a user-configured threshold.

+-------------------------------------------------------------+
|                     Vertex AI Endpoint                      |
|  [ Live Serving Requests ] ----> [ Vertex AI Model ]        |
+------------------------------------+------------------------+
                                     | (Request-Response Logs)
                                     v
                  +--------------------------------------+
                  |   Vertex AI Model Monitoring         |
                  +--------------------------------------+
                         |                      |
      (Compare vs. Baseline)          (Compare Rolling Windows)
                         v                      v
                [ Skew Detection ]      [ Drift Detection ]
                         |                      |
                         +----------+-----------+
                                    | (Distance > Threshold)
                                    v
                          [ Cloud Monitoring Alert ]
                                    |
                                    v
                        [ Vertex AI Retraining ]

Infrastructure Observability, Logging, and Automated Troubleshooting

Infrastructure observability and model logging provide the diagnostic data necessary to isolate errors across the machine learning serving lifecycle. An endpoint writes operational metrics—including CPU utilization, GPU memory consumption, request latency, and HTTP error codes—directly to Cloud Monitoring. Enabling request-response payload logging exports raw prediction inputs and outputs to BigQuery tables for offline analysis, auditing, and continuous evaluation against delayed ground-truth labels. When feature drift or accuracy loss is detected, automated workflows triggered through Vertex AI Pipelines run data validation checks, trigger retraining jobs on updated datasets, and deploy the resulting candidate model using a canary traffic split to verify stability before full rollout.

Key Facts

• Training-serving skew occurs when the distribution of data used during training differs from the data presented to the model during production inference.
• Vertex AI Model Monitoring detects feature skew by comparing serving requests to training data baselines, and it detects feature drift by comparing recent serving windows to historical serving distributions.
• Vertex Explainable AI provides feature attributions using methods like Shapley values and Integrated Gradients to identify which inputs drive predictions and uncover hidden bias.
• Request-response logging records production prediction payloads from Vertex AI endpoints to BigQuery for auditing, debugging, and continuous evaluation.