Prepare and test your skills
Prepare and test your skills
Vertex AI Model Monitoring (opens in a new tab) on the Gemini Enterprise Agent Platform is the continuous evaluation system that detects drift in production models and triggers operational alerts. The system samples serving request-response payloads from an Agent Platform Endpoint, compares live distributions and predictions against a training baseline, evaluates quantitative thresholds, and exports violations to Cloud Monitoring (opens in a new tab) and Pub/Sub for real-time response. This closed loop turns statistical drift into an actionable incident before predictive performance collapses.
Training-serving skew and inference drift are the two distribution comparisons that anchor drift detection in Vertex AI Model Monitoring. Training-serving skew compares the feature distribution in live serving requests against the feature distribution in the training dataset that produced the deployed Model resource. Inference drift compares feature and prediction distributions in a recent production window against distributions in an earlier production window, which reveals change even when training data is no longer representative.
The monitoring job depends on the serving layer to log a representative sample of request-response payloads to a data store for regular analysis. The job supports online prediction traffic from an Endpoint and batch prediction jobs for tabular AutoML models, custom tabular models including TensorFlow Serving and XGBoost containers, AutoML image models, and BigQuery ML models. For each sampled payload, the job extracts configured input features and output predictions, computes distribution statistics over the monitoring window, and compares them to baseline statistics using a distance measure. When an upstream application change causes serving features to diverge, such as sending a product name instead of a product code, the detector flags the inconsistency as training-serving skew rather than natural evolution.
Quantitative thresholds in Vertex AI Model Monitoring convert statistical distance into a binary alert decision for each monitored feature and prediction. A threshold policy applies to a specific Monitoring Job, which owns the baseline dataset, the sampled serving data, the list of skewed and drifted attributes, and the alert condition evaluated in each window. The policy defines what distance is tolerable for a given feature type, how long a violation must persist, and which features trigger an alert versus informational logging. Setting thresholds per critical feature prevents noisy low-importance fields from masking drift in revenue-critical inputs.
Three drift classes require distinct thresholds and baselines:
Exam tip: Training-serving skew compares serving data to training data, while inference drift compares recent serving data to earlier serving data; use skew to catch pipeline bugs and drift to catch changing world behavior.
Response quality degradation is the decline in usefulness, safety, and correctness of production predictions even when raw feature distributions appear stable. Vertex AI Model Monitoring tracks quality through continuous evaluation metrics computed from sampled request-response pairs, delayed ground-truth labels when available, and attribution scores from Vertex Explainable AI. The monitoring job joins predictions with labels on arrival, recomputes accuracy, precision, recall, calibration, and custom safety metrics per window, and compares the result to validation metrics stored in Vertex ML Metadata and the Model Registry. Attribution drift adds a diagnostic layer by showing when the ranking of important features changes, which often precedes an accuracy drop.
Continuous evaluation pipelines sustain measurement by versioning every element that affects quality. The pipeline tracks lineage from source code and pipeline components to pipeline runs, datasets, transformed schemas, model checkpoints, and evaluation results, so a regression can be traced to a specific data, code, or configuration change. Conditional pipeline steps must emit observability signals when key steps do not run or fail, which prevents silent gaps in quality measurement. When quality metrics breach satisficing thresholds for latency, size, accuracy, or safety in staging or canary serving, the pipeline holds promotion and preserves fast rollback to the previous Model version on the Endpoint.
Automated alerting with Cloud Monitoring and Pub/Sub turns a threshold breach into a real-time operational incident owned by the serving team. Vertex AI Model Monitoring emits a metric time series for each skew, drift, and quality signal to Cloud Monitoring, where an alerting policy evaluates the threshold condition over each monitoring window. When the condition is met, traffic flows from the alerting policy to a Pub/Sub topic, and data flows from the Pub/Sub topic to each subscriber such as a retraining pipeline, a ticketing function, or an on-call notification channel. The Endpoint continues to serve traffic on one side of the trust boundary while monitoring, alerting, and remediation operate on the observability side, so monitoring failures never block predictions.
The end-to-end detection and alerting lifecycle runs in a fixed order:
Choose batch-oriented monitoring windows for high-volume tabular and batch prediction workloads to reduce noise, and choose short, sensitive windows with Pub/Sub push delivery for generative, safety-critical, or low-volume high-risk Endpoints where minutes matter. Tighter thresholds increase recall of real incidents but raise operational cost from false positives, while looser thresholds reduce paging load but allow silent staleness as serving data evolves away from learned patterns.
Diagnose Agent Quality Degradation and Orchestrate Remediation Workflows is the practice of detecting production quality loss, tracing the failure to data, code, or serving causes, and routing the fix through validated release and retraining pipelines. On the Gemini Enterprise Agent Platform (opens in a new tab), monitoring checks live serving behavior for skew and drift, lineage and payload logs provide the trace, and Agent Platform Pipelines (opens in a new tab) orchestrates the corrective run from validation through canary release to full rollout.
Continuous monitoring of serving effectiveness detects when a deployed model no longer provides accurate predictions. Model Monitoring on Gemini Enterprise Agent Platform addresses two distinct failure modes: training-serving skew and statistical drift leading to staleness. Training-serving skew is an inconsistency introduced between serving features and the features expected by the model, for example a recommendation model expecting an alphanumeric most-recently-viewed product code but receiving the product name instead after an update to the consuming application. Statistical drift is a change over time in the properties of serving data so patterns learned by the current model are no longer accurate.
The documented control is to log a sample of serving request-response payloads in a data store for regular analysis, which creates the production evidence needed to distinguish a sudden skew break from gradual drift.
Exam tip: Training-serving skew is a feature mismatch at serve time, such as wrong field or format, while drift or staleness is a shift in data distribution over time; both cause predictive performance to degrade.
Traceability on the Gemini Enterprise Agent Platform links a production symptom back to the exact code, data, and pipeline run that produced the model. Vertex ML Metadata and the Model Registry on Gemini Enterprise Agent Platform store that history when teams track lineage of source code, deployed pipelines, pipeline components, pipeline runs, the dataset in use, and the produced artifacts. Teams also track hyperparameters and pipeline-run configurations, plus key step inputs and outputs such as dataset statistics, dataset anomalies, transformed data and schemas, model checkpoints, and model evaluation results.
Agent Platform Pipelines is a serverless service that runs Kubeflow pipelines as a directed acyclic graph, where each step is a component instance with inputs, outputs, and a container image and step inputs can depend on outputs of other steps. Because dependencies define execution order, a failed or skipped conditional step can be isolated by checking whether the step ran in response to its condition. The guidance is to ensure observability by adding alerting mechanisms in case key steps do not run or fail, and to inspect components in the pipelines graph interface for transformed tables, evaluated architectures, process status, and logs.
Release validation proves that a model artifact is fit to serve in its target environment before it receives broad live traffic. The model artifact is first staged in a sandboxed version of the serving environment to verify that the artifact can be loaded and invoked with its runtime dependencies and that required operations and binaries are present. In staging, teams validate satisficing metrics such as model size and latency, unit-test model-artifact-serving interfaces against raw inputs and expected outputs, and unit-test the artifact on typical and edge cases such as a request where all features are set to None. If variations of the same model will deploy to different serving environments or workloads, such as cloud versus on-device or real-time versus batch serving, validation against the satisficing metric is deferred because each target may require a different threshold.
After deployment, release proceeds in controlled exposure order:
Remediation pipelines rebuild quality by retraining or tuning on corrected data and promoting the result through the same tracked workflow. Agent Platform Pipelines orchestrates this work because dependencies between steps define the workflow as a directed acyclic graph, so data splitting, transformation, training, and evaluation run in order and each step consumes artifacts or primitives from prior steps. Tuning support is provided in the tuning pipelines for BERT, T5, and image-generation (GPU), and Gemini models support supervised tuning from multimodal datasets. For multimodal tuning, the assemble method builds a new BigQuery table where each row contains the request body with contents and systemInstruction parts, including fileData with fileUri and mimeType for images, which standardizes training rows before the tuning job runs.
Before committing to a full tuning run, teams validate and size the job with two assessments. The validity assessment checks whether the dataset contains errors such as dataset formatting errors or model errors by calling assess_tuning_validity() in the Agent Platform SDK or the assess method with a TuningValidationAssessmentConfig in REST, specifying model name such as gemini-3.5-flash and dataset usage such as SFT_TRAINING. The resource assessment estimates token and billable character count by calling assess_tuning_resources() in the SDK or the assess method with a TuningResourceUsageAssessmentConfig in REST, which returns counts for cost and capacity planning.
Execution troubleshooting maps an API or queue symptom to a system cause and a defined mitigation. Client runners, evaluation loops, and program queues surface HTTP statuses that distinguish bad requests, permissions, timeouts, quota exhaustion, and backend overload. The documented matrix preserves the cause on the platform side and the action on the client side, including strict timeouts, credential checks, and retry behavior with backoff.
| HTTP status | Error type | System cause and mitigation |
|---|---|---|
| 400 | INVALID_ARGUMENT | Missing evolve-block markers, invalid JSON schema syntax, or exceeded file or lines-of-code limits; verify block comments are inside function bodies and file-count and bounds are respected |
| 403 | PERMISSION_DENIED | Missing Discovery Engine User role or Gemini Enterprise license; verify licenses and configure application default credentials with gcloud auth application-default login |
| 408 | LOCK_TIMEOUT | Client runner exceeded execution duration and program lock expired; enforce 30-minute client-side timeouts and submit failure penalty scores to clear the queue lock |
| 429 | RESOURCE_EXHAUSTED | Exceeded concurrent execution limits or model quotas; use client-side exponential backoff and conservative concurrency in RunSettings |
| 503 | SERVICE_UNAVAILABLE | Backend overloaded or in maintenance; retry with randomized exponential backoff on the client side |
A related silent failure is a safety filter intercept, or silent drop, where mutated candidate prompts are flagged and dropped for sensitive phrasing, the queue returns empty responses, and runners may wait indefinitely. Remediation is to sanitize the problem description to remove emotionally charged or security-sensitive phrases, parse and truncate raw exception traces or terminal stderr logs in insights payloads to avoid echoing unsafe content, and localize datasets by loading training records or large corpora locally in the client evaluation loop instead of placing them in prompts.
Continuous evaluation pipelines on the Gemini Enterprise Agent Platform are automated workflows that repeatedly capture live production traffic and score it against baseline datasets to detect drift and maintain reliability. Vertex AI Pipelines (opens in a new tab) runs these workflows as serverless Kubeflow pipelines where each step has inputs, outputs, and a container image, and step dependencies define the workflow as a directed acyclic graph. Model Monitoring on Gemini Enterprise Agent Platform (opens in a new tab) provides the monitoring framework, with related capabilities in Vertex Explainable AI and model monitoring with BigQuery ML, while evaluated models are stored in Model Registry on Gemini Enterprise Agent Platform.
Automated evaluation schedules are recurring Vertex AI Pipelines runs that automate and monitor machine learning and data preparation tasks for production agents. The pipeline depends on captured inference data, so the serving system must log a sample of serving request-response payloads into a data store for regular analysis. Data then flows from the data store to the evaluation steps, where live interactions are compared to baseline datasets derived from training and evaluation splits. Because steps can depend on the output of other steps, the capture, transformation, scoring, and alerting stages execute in order as a directed acyclic graph.
Continuous evaluation metrics on Vertex AI divide into rule-based checks and model-based judgments for quality, safety, and performance. Rule-based metrics apply deterministic thresholds such as model size, latency, valid output format, and optimizing versus satisficing metric targets. Model-based metrics use an evaluator model to score groundedness, safety, and coherence of live agent responses against grounded reference data. Baselines are established from versioned evaluation datasets, dataset statistics, transformed data and schemas, model checkpoints, and prior evaluation results.
Baseline thresholds must account for the target serving environment because different environments require different satisficing metric thresholds. A continuous training pipeline may train two or more models where each model is optimized for its target deployment environment, such as cloud environments versus on-device environments or real-time environments versus batch serving environments. When variations of the same model will serve different workloads, validation against the satisficing metric is deferred from training to the model deployment phase and performed in a staging version of each serving environment.
Training-serving skew detection identifies an inconsistency introduced between serving features and the features expected by the model. For example, a recommendation model expects an alphanumeric product code for a feature like most-recently-viewed product code, but the product name is passed during serving after an update to the application that consumes the model service. Drift detection identifies a separate failure mode where the statistical properties of serving data change over time, so patterns learned by the deployed model are no longer accurate. In both cases the deployed model can no longer provide accurate predictions, which is why continuous monitoring must regularly and proactively verify that effectiveness does not degrade.
Exam tip: Skew is a feature mismatch at serving time, such as receiving a product name instead of a product code, while drift is a change in data statistics over time that makes a previously accurate model stale.
Staged validation verifies that a model artifact is fit to serve before continuous monitoring begins in production. Validation starts in a sandboxed staging environment and moves to live traffic only after each gate passes:
None.The three live-traffic gates differ by scope and decision:
| Gate | Traffic scope | Decision supported |
|---|---|---|
| Smoke test | Single instance or batch sent to deployed service | Service responds correctly in target environment |
| Canary test | Small stream of live serving data | New version is error-free before broad exposure |
| Online experimentation | Small subset of serving population with A/B testing or MAB | New model outperforms current model and should fully release |
Vertex ML Metadata is the tracking system that makes continuous evaluation reproducible by linking source code, pipelines, runs, datasets, and produced artifacts. As more continuous-training pipelines with complex workflows enter production, lineage tracking allows teams to trace production issues, debug failures, and reproduce pipeline outputs for the next development iteration. Tracking also provides observability for conditional pipeline steps by recording whether steps ran in response to conditions and adding alerting when key steps do not run or fail.
Track the following for every evaluation and training run:
Standard artifact types defined by Google Cloud Pipeline Components carry these references between steps, including google.VertexModel for a Gemini Enterprise Agent Platform Model resource, google.VertexEndpoint for an Endpoint resource, google.VertexBatchPredictionJob for a batch prediction job, google.VertexDataset for a Dataset resource, google.BQMLModel for a BigQuery ML model, google.BQTable for a BigQuery table, and google.UnmanagedContainerModel for an unmanaged container model. A component consumes an artifact by reading its metadata, such as the resourceName, and produces artifacts through outputs or an importer node that registers the artifact URI and metadata.
Training-serving skew compares the feature distribution in live serving requests against the feature distribution in the training dataset that produced the deployed Model resource, while inference drift compares feature and prediction distributions in a recent production window against distributions in an earlier production window. Use skew to catch pipeline bugs such as sending a product name instead of a product code and use drift to catch changing world behavior when training data is no longer representative.
Vertex AI Model Monitoring emits a metric time series for each skew, drift, and quality signal to Cloud Monitoring, where an alerting policy evaluates the threshold condition over each monitoring window. When the condition is met, Cloud Monitoring publishes a message with job ID, feature name, distance score, and window to a Pub/Sub topic, which delivers it to subscribers such as a retraining pipeline, a ticketing function, or an on-call notification channel.
Validation starts with staging load and invoke checks, satisficing validation for size and latency, and interface and edge-case unit tests including a request where all features are set to None. After deployment it proceeds to a smoke test with a single instance or batch to validate the response, a canary test with a small stream of live serving data to confirm no errors, and online experimentation on a small subset with A/B testing or Multi-Armed Bandit (MAB) to decide on full release, with rollback to the previous Model version verified in staging.
Last updated