Prepare and test your skills
Prepare and test your skills
An automated evaluation pipeline is a repeatable workflow that runs the same metrics on the same validation and test sets for every model version, then continues to watch that model in production. In Google Cloud, Agent Platform Pipelines (opens in a new tab) orchestrates the evaluation steps, Model Monitoring on Agent Platform (opens in a new tab) tracks drift and skew in serving data, and Cloud Monitoring (opens in a new tab) fires alerts when metrics degrade. Together they create a closed loop where evaluation results flow from datasets to pipeline components to monitoring dashboards, and monitoring alerts flow back to retraining or rollback actions.
Standardized evaluation starts with indicators that separate training optimization from business success. Computation-based metrics are calculated by numeric formulas, while qualitative indicators such as fluency or entertainment value cannot be computed directly and require LLM-as-a-judge or human labeling. Training loss can differ from the business metric, so a fraud model may train with cross-entropy loss but be evaluated on recall because recall measures coverage of fraudulent transactions. For traditional models, Agent Platform provides recall, precision, and area under the precision-recall curve (AuPRC), plus accuracy, F1-score, AUC-ROC, and log loss.
Evaluation data must be large and varied enough to be statistically significant. Three complementary sources establish that significance:
Data for the Agent Platform evaluation service is stored in BigQuery or Cloud Storage. BigQuery supports SQL-based inference tasks such as natural language processing and machine translation, while Cloud Storage provides cost-efficient storage for large evaluation datasets.
Agent Platform Pipelines, built on Kubeflow Pipelines, standardizes evaluation by packaging it as a versioned pipeline component that runs alone or with training and deployment steps. The evaluation component loads a model version and a fixed dataset, computes the defined metrics, compares versions against thresholds such as minimum accuracy, maximum error rate, or minimum F1-score, and logs parameters, code versions, dataset versions, and results to Agent Platform ML Metadata and Experiments on Agent Platform. Because results and lineage flow from the pipeline to ML Metadata to Model Registry, teams can trace which dataset and code produced which model and revert to the most recent stable version.
Building a reproducible process follows an ordered sequence:
Once standardized, the pipeline supports continuous training. A pipeline can run on a schedule or be triggered by new data or by a monitoring alert, run custom training or foundation-model fine-tuning, register the artifact in Model Registry or Cloud Storage, and promote the model only after it passes evaluation thresholds.
Model Monitoring on Agent Platform tracks how serving data diverges from expectations after deployment. The service compares live input and output distributions against a baseline, which can be the training set analyzed with TensorFlow Data Validation (TFDV) or past production traffic stored in the golden dataset. Model Monitoring on Agent Platform v2 can monitor models on a managed Agent Platform endpoint and on self-hosted serving on Google Kubernetes Engine (GKE), and it can track metrics at the model level rather than only at the endpoint level for granular performance tracking.
Skew and drift answer different questions and depend on different baselines:
| Approach | Baseline compared | Use when |
|---|---|---|
| Skew detection | Production data vs training data | Training data is available; distortion between training and serving indicates expected performance no longer holds |
| Drift detection | Production data vs earlier production data over time | Training data is unavailable; inputs changing over time signal degrading accuracy |
Model Monitoring supports three v2 objectives: input feature drift, output drift, and feature attribution change. Feature attribution from Vertex Explainable AI is an early indicator of degradation when the model starts relying on different features than it used in training and testing, which is especially useful for embeddings and time series that are hard to compare directly. Monitoring applies to structured numerical and categorical features, not to unstructured data such as images. For batch prediction, the same approach applies by running standard jobs in the Agent Platform evaluation service and enabling monitoring for batch inferences, while batch resource use is observed in Metrics Explorer.
Exam tip: Choose skew detection when you can point the monitoring job at training data; choose drift detection when training data is unavailable and you must watch production data change over time.
The Gen AI evaluation service (opens in a new tab) extends continuous monitoring to generative outputs in production. The service enables automatic metrics for response quality, safety, instruction adherence, grounding, writing style, and verbosity, plus toxicity scores and user-feedback ratings. During testing, its grounding metric checks factuality against provided source text, and human-in-the-loop review assesses quality, relevance, safety, and guideline adherence. Model Armor (opens in a new tab) can additionally screen model outputs for unsafe, biased, off-topic, or malicious content.
Serving health is tracked separately from output quality. Gemini models expose built-in metrics such as time to first token (TTFT) in Metrics Explorer, and the model observability dashboard shows model queries per second, token throughput, first-token latency, overall latency, throughput, and error rates. Cloud Monitoring tracks request rate, error rate, latency, and resource utilization for Agent Platform endpoints, while Cloud Logging collects logs from all pipeline and serving components for troubleshooting. For generative endpoints, SLO targets make degradation concrete, such as 99.9% successful API calls, 95th percentile inference latency below 300 ms, TTFT below 500 ms for 99% of requests, and harmful-output rate below 0.1%.
Alerting turns monitoring signals into action by applying custom thresholds to Model Monitoring and Cloud Monitoring metrics. Thresholds depend on the use case, domain expertise, and initial monitoring baselines, and they are configured as Cloud Monitoring alert policies with custom dashboards and log-based metrics from Cloud Logging. When Model Monitoring detects significant skew, drift, or prediction anomalies, or when Model Armor flags unsafe outputs, alerts flow from Cloud Monitoring through Pub/Sub integrations to notification channels such as Slack, email, SMS, or PagerDuty, customized by severity and owning team.
The alert then triggers a defined remediation path that depends on the pipeline and registry. A feedback loop starts an Agent Platform Pipelines retraining job automatically when an alert fires, and insights from monitoring continuously improve the next model version. If the new version fails evaluation or production SLOs, the team uses lineage in Model Registry to roll back to the last stable version, disables a problematic endpoint, pauses a misbehaving pipeline, or isolates a compromised dataset in BigQuery or Cloud Storage.
Selecting and applying model evaluation metrics means matching the metric to the prediction type and the business risk of being wrong. Predictive tasks use scores that compare predictions to labeled ground truth, while generative tasks use quality scores that judge the usefulness and truthfulness of free-form responses. On Google Cloud, both kinds of scores can be produced by automated evaluation pipelines that store prompts, responses, and scores for offline analysis.
The F1 Score (opens in a new tab) is a metric used to evaluate the accuracy of a model's output when both precision and recall are important. The F1 Score balances missed positives against false positives in a single score, so a model cannot score well by favoring one at the expense of the other. The documentation calls out information extraction as a task where this balance matters, because both finding all relevant facts and avoiding incorrect facts affect quality. Choose the F1 Score when the business objective penalizes both types of errors, such as extracting entities, defects, or clinical terms where omission and over-prediction are both costly.
Generative quality metrics assess responses produced by the serving subsystem rather than single-label predictions. The quality evaluation subsystem produces evaluation scores for metrics like factual accuracy and relevance by testing the serving subsystem with evaluation prompts. Factual accuracy measures whether the generated response is grounded in retrieved domain data, which matters for clinical assistance grounded in electronic health record data or PubMed and for legal research grounded in statutes and case records. Relevance measures whether the response answers the user's request in context, which matters for personalized product recommendations augmented with buying behavior and reviews.
| Task type | Example metric from documentation | Choose when |
|---|---|---|
| Predictive classification and extraction | F1 Score | Both precision and recall matter, such as information extraction |
| Generative RAG response | Factual accuracy, relevance | Generated text must be grounded and useful, such as clinical, legal, or shopping assistants |
Exam tip: Use F1 Score when both precision and recall matter for predictive output, and use factual accuracy and relevance when judging generated RAG responses.
The automated quality evaluation subsystem on Google Cloud is the pipeline that turns evaluation prompts into stored quality scores. An evaluation, or eval, is an experiment that sends logged or synthetic queries through an experimental stack and a base stack to produce diffs and metrics on quality and user-experience impact. The production implementation decouples evaluation from serving so scoring can run continuously without blocking user traffic.
Evaluation data flows from storage through compute to analysis in an ordered sequence:
The pipeline depends on the data ingestion subsystem for prompts and on the serving subsystem for responses, while BigQuery depends on the function output for offline analytics. Because scores, prompts, and responses land together in BigQuery, teams can track factual accuracy and relevance over time, compare changes, and justify whether a model update improves business-relevant quality.
LLM-as-a-Judge is an automated evaluation pattern where a high-quality foundation model, such as Gemini (opens in a new tab), scores or compares the outputs of candidate generative models for qualities like coherence, relevance, and safety. AutoSxS (opens in a new tab) is the Google Cloud implementation of this pattern that runs side-by-side comparisons of two models and benchmarks the automatic judge against systematic human evaluation. Together they solve the core generative AI evaluation problem where prompts and outputs are too complex and high-dimensional for string-overlap metrics alone.
A custom evaluation dataset is a curated set of prompts and contexts that reflects essential, average, and edge cases for a specific application. The dataset is required because the pre-training distribution of a foundation model is typically unknown and generic metrics such as BLEU (opens in a new tab) for translation and ROUGE for summarization cannot capture multi-instruction compliance, factual grounding, or nuanced text quality. Powerful large language models (LLMs) can generate, curate, and augment this dataset to cover rare scenarios quickly, but humans must still check the synthetic data for quality assurance. In a retrieval-augmented generation (RAG) application this curation extends to versioned prompt templates, rewritten user queries, and vector databases with embeddings and chunking strategies that must all be tracked reproducibly.
Manual review works during prototyping when developers inspect a few outputs for qualitative feel, but manual review becomes a bottleneck as test cases grow. Automated evaluation moves faster, improves reliability, and removes individual subjectivity so results are reproducible across runs. The tradeoff is that automation must itself be validated, which is why human-preference data is retained as ground truth for comparison.
AutoSxS is an evaluation pipeline that compares Model A against Model B by running inference on both models and then asking an autorater LLM to choose the better response. The pipeline can evaluate a tuned first-party model from Vertex Model Registry against a reference model from the same registry, with inference configured to run on both models simultaneously. The behavior of the autorater is controlled by autorater prompt parameters that define the judging criteria, such as coherence, relevance, helpfulness, and safety, plus the output location for results.
An AutoSxS run flows through an ordered sequence of components:
Results are written to Cloud Storage in JSONL format or to a BigQuery table for row-level analysis.
Judgments are the example-level outputs of AutoSxS that explain model performance one prompt at a time. Each judgment preserves the full evaluation context so a reviewer can audit why the autorater preferred one response without rerunning inference. A judgment combines the inference prompt, both model responses, the autorater decision, a natural-language rating explanation, and a confidence score.
When stored in BigQuery, the judgments table uses a consistent schema where each row is one comparison:
inference_instruction is the instruction used to generate both model responses.inference_context is the additional context, such as retrieved grounding data, supplied with the instruction.response_a is Model A's response to the instruction and context.response_b is Model B's response to the same instruction and context.choice is the autorater decision with values Model A, Model B, or Error, where Error means a failure prevented a decision.confidence is a score between 0 and 1 that signifies how confident the autorater was in its choice.explanation is the autorater's reason for its choice.Confidence and explanation turn a win into auditable evidence, because a high win rate with low confidence or with explanations that miss the intended criteria signals a misconfigured autorater prompt rather than a better model.
Aggregate metrics are dataset-level summaries calculated from the judgments table that report how often each model won. When no human-preference data is provided, AutoSxS produces only the AutoRater Model A win rate and the AutoRater Model B win rate, each defined as the percentage of examples where the autorater chose that model. These win rates answer which model the judge preferred overall, but they do not prove correctness on their own. Reviewers must inspect row-based results and explanations to confirm that the wins align with expectations for coherence, factuality, and safety before promoting a tuned model.
Human evaluation is the systematic collection of human preferences on the same instruction-context-response triples to serve as ground truth for the autorater. When human-preference data is supplied, AutoSxS outputs both sets of win rates plus a discrepancy analysis that treats Model A as the positive class and measures how often the autorater agrees with people. Accuracy reports raw agreement percentage, while precision, recall, and F1 quantify the autorater's ability to detect Model A wins from the autorater's perspective, the human perspective, and their harmonic mean.
The four agreement counts define every possible outcome and support the derived metrics:
| Outcome | Autorater choice | Human preference |
|---|---|---|
| TP | Model A was better | Model A was better |
| FP | Model A was better | Model B was better |
| TN | Model B was better | Model B was better |
| FN | Model B was better | Model A was better |
Cohen's Kappa (opens in a new tab) is the chance-corrected measure of agreement between autorater and human raters that accounts for the likelihood of random agreement. Kappa ranges from -1.0 to 1.0, where values at or below 0.0 indicate agreement no better than chance, 0.0 to 0.2 is slight, 0.2 to 0.4 is fair, 0.4 to 0.6 is moderate, 0.6 to 0.8 is substantial, 0.8 to 1.0 is nearly perfect, and 1.0 is perfect agreement. A high win rate with low accuracy or low Kappa means the autorater diverges from human judgment and its prompt, criteria, or model choice must be revised before the automated scores can drive iteration.
Exam tip: A high AutoRater win rate does not validate a model by itself; confirm alignment with human-preference accuracy and Cohen's Kappa and read the per-example explanations before acting on the result.
Skew detection compares production data against training data to detect distortion indicating expected performance no longer holds, while drift detection compares production data against earlier production data over time to detect changing inputs. Choose skew detection when training data is available and drift detection when training data is unavailable and you must watch production data change over time.
Use the F1 Score for predictive classification and extraction tasks such as information extraction where both precision and recall matter because omission and over-prediction are both costly. Use factual accuracy and relevance for generative retrieval-augmented generation (RAG) responses such as clinical, legal, or shopping assistants where generated text must be grounded in retrieved domain data and useful in context.
AutoSxS runs inference on both Model A and Model B for the same prompts and then asks an autorater large language model (LLM) to choose the better response with a choice, confidence score, and explanation. The judgments are aggregated into AutoRater Model A and Model B win rates, and when human-preference data is supplied the autorater must be validated against it with accuracy and Cohen's Kappa before acting on the result.
Last updated