Evaluating Model Garden Models for Agent Tasks

Analyzing task requirements and mapping them to model capabilities is the process of breaking a business problem into precise functional needs and selecting a Gemini Enterprise Agent Platform Model Garden (opens in a new tab) model that satisfies them. The candidate model receives input prompts and produces responses, the Gen AI evaluation service (opens in a new tab) measures those responses against defined criteria, and the deployment configuration determines whether latency, throughput, cost, and accuracy constraints are met. A model is an optimal fit only when its supported task, input and output handling, and serving compatibility all align with the use case.

1.1Deconstructing the Target Task into Functional Requirements

Deconstructing the target task means translating a use case such as summarizing news articles, answering customer inquiries, or assisting with code writing into a specific objective, input shape, and expected output. The objective defines what the model must predict, such as classification, regression, summarization, translation, or generation, and the evaluation target column names the column from the training data that the model is trained to predict. Input requirements include the prompt dataset stored in BigQuery or Cloud Storage and the ground truth that represents the expected result, while output requirements include the batch prediction output stored as a BigQuery table or Cloud Storage URI. This decomposition lets an architect compare each requirement directly against a model card instead of selecting by popularity alone.

Model Garden organizes functional requirements around supported pipeline tasks that define modality and behavior. The evaluation metrics (opens in a new tab) for each task depend on this mapping:

• text-generation for open-ended generation and conversational responses
• text2text-generation for transformations such as summarization and translation
• text-to-image for generating images from text prompts
• feature-extraction for producing embeddings for search and similarity
• sentence-similarity for comparing semantic closeness between texts
• image-text-to-text for vision-language tasks that combine image and text inputs

If a task requires a modality or pipeline task outside this set, the model is not fully supported for one-click deployment in Model Garden.

1.2Mapping Requirements to Model Attributes and Serving Compatibility

Mapping requirements to model attributes means checking that the model accepts the required input modalities, produces the required output form, and runs in a supported serving container. A text-only classification task maps to a text model with a classification objective, while a summarization task over long articles requires a model whose input handling covers long prompts and whose output handling supports multi-sentence responses. A vision task that takes images plus text maps specifically to an image-text-to-text model, and an embedding or retrieval task maps to feature-extraction or sentence-similarity rather than a generative model.

Serving compatibility is a hard constraint in the mapping. The vLLM container supports text-generation and image-text-to-text models, so a model outside those tasks or outside the supported container cannot use the default one-click path even if its capabilities look correct. When a desired model is missing from Model Garden, the documented reasons are that it is not a top trending model with strong community interest, it is not yet compatible with a supported serving container, or it uses an unsupported pipeline task. In that case the workflow depends on alternative paths: deploy the model directly with a custom Colab notebook (opens in a new tab), submit a feature request through Model Garden or support channels, or monitor Model Garden for future additions.

1.3Validating Fit with Evaluation Metrics

Validating fit means defining criteria and metrics before selection and then measuring candidate outputs against them. Criteria are the dimensions to evaluate, such as conciseness, relevance, correctness, or appropriate choice of words, while a metric is a single score that measures model output against those criteria. The Gen AI evaluation service provides model-based metrics, where a judge model assesses output against descriptive criteria, and computation-based metrics, where mathematical formulas compare output against ground truth or reference text such as ROUGE and BLEU.

Model-based evaluation moves data through an ordered sequence:

1. Data preparation: Input prompts flow from the evaluation dataset to each candidate model, and each candidate model generates corresponding responses.
2. Evaluation: Generated responses and evaluation metrics flow from the evaluation service to the judge model, and the judge model evaluates each response individually.
3. Aggregation and explanation: Row-based judgments flow from the judge model to the evaluation service, which aggregates them into an overall score with chain-of-thought explanations for each judgment.

The judge model for most uses is Gemini, with MetricX or COMET available for translation, and proprietary Google judge models are calibrated with human raters and managed out of the box. The choice between metric families depends on data availability, cost, and speed:

Within model-based metrics, pointwise metrics let the judge model score one candidate response on a scale such as 0 to 5, while pairwise metrics let the judge model compare responses from two models and pick the better one. Pairwise comparison is used for candidate versus baseline decisions and is only supported with Gemini as the judge model.

1.4Tuning Deployment Configuration to Task Constraints

Tuning deployment configuration means moving from a baseline one-click deployment to a tailored serving setup that balances latency, throughput, cost, and accuracy for the specific task. The default deployment configuration cannot satisfy every combination of those priorities, so the documented pattern is to first experiment with one-click deployment to establish a baseline and then fine-tune using a Colab notebook or Python SDK. Data flows from the Model Garden model to the selected serving stack, and the serving stack determines how requests are batched, scaled, and accelerated under load.

The custom deployment paths correspond to different inference servers and frameworks, including vLLM, TGI, TEI, and HF PyTorch inference notebooks. Choosing among them depends on whether the task is generative text serving, embedding serving, or custom PyTorch logic, and on the throughput and cost tradeoffs observed in the baseline. This iterative tuning ensures that a model selected for functional fit also meets operational constraints before promotion to production.

Evaluating model performance metrics and operational constraints is the practice of comparing candidate Model Garden models on latency, throughput, accuracy, and cost, then matching the winner to a deployment target that can sustain it in production. A low-code agent solution (opens in a new tab) depends on both model quality and operational feasibility, including accelerator availability, scaling behavior, input delivery, and continuous evaluation. The selection decision therefore combines serving metrics, hardware and cost controls, inference routing, and pairwise quality measurement.

2.1Operational Telemetry for Latency, Throughput, and Utilization

Operational telemetry for Model Garden models is the continuous collection of serving and hardware signals that reveal whether a model meets latency, throughput, and cost requirements. Cloud Monitoring collects utilization and serving signals, Cloud Logging (opens in a new tab) collects logs, and analysis in BigQuery or Looker Studio forecasts future demand for graphics processing units (GPUs) and tensor processing units (TPUs) from growth or new models. The model observability dashboard exposes model queries per second (QPS), token throughput, and first token latencies for generative AI serving. Proactive management validates capacity estimates with load testing using Apache JMeter or LoadView, tests generative serving under high concurrent load to find where accelerator availability limits performance, and sets proactive alerts for agent quality regressions.

Key signals to compare across candidate models include:

• Latency, including first token latency for streaming generative responses
• Throughput, including model QPS and token throughput under concurrent load
• Errors and resource utilization, especially GPU and TPU utilization
• Historical throughput rates and usage trends used to forecast accelerator demand

Capacity planning flows from telemetry to action: monitoring data predicts when and where specialized accelerators are needed, load tests confirm behavior under stress, and resource quotas are increased before production demand exceeds obtainable capacity.

2.2Online Versus Batch Inference Deployment Options

Online versus batch inference deployment is the choice between near real-time scoring for live applications and scheduled scoring at a regular cadence. Vertex AI (opens in a new tab) online prediction receives input instances and returns predictions in the response, while batch prediction passes input data to a cloud-hosted model and returns inferences for each data instance. Online callers first fetch features from a low-latency store, such as a Google Cloud database or the Feature Store online serving API, then call the prediction endpoint with those inputs. Batch callers fetch data from a data lake or the Feature Store batch serving API, while streaming ingestion makes updated feature values available for online serving within seconds.

The endpoint design also affects operational cost and evolution. Custom prediction routines customize request and response handling to and from the model server, multiple models can share one endpoint to gradually replace a model, and separate endpoints can isolate testing from production while sharing resources across deployments.

2.3Compute Selection and Cost Controls for Accelerators

Compute selection for inference and training is the matching of central processing unit (CPU) virtual machine (VM) types and GPU types to model size, latency targets, and budget. Committed use discounts (CUDs) reduce cost for stable 24x7 inference or training with fixed or predictable capacity on VMs and accelerators. Spot VMs lower cost for fault-tolerant evaluation and experimentation workloads but can be preempted, while Dynamic Workload Scheduler (DWS) improves obtainability of high-demand accelerators. DWS Flex-Start mode supports complex batch training that needs high-end GPUs for up to seven days, and Calendar mode reserves specific H100 and H200 GPUs and Trillium TPUs for longer workloads up to three months.

Advanced serving abstracts fluctuating capacity and topology complexity. A vLLM engine dynamically uses TPUs and GPUs to address fluctuating inference capacity and performance needs, Cluster Director deploys and manages accelerator groups with colocation and scheduling for multi-GPU training on A3 Ultra H200 and A4 B200 across GKE and Slurm clusters, and Ray on Vertex AI abstracts distributed infrastructure so applications request resources for training and serving without directly managing VMs and containers.

2.4Automatic Scaling and Load Distribution for High Availability

Automatic scaling and load distribution keep inference highly available and low latency as demand fluctuates. GKE Inference Gateway (opens in a new tab) deploys models behind a load balancer with model-aware routing, so traffic distributes across multiple instances based on model metrics, optimizes resource utilization, and preserves user experience. Vertex AI online prediction turns on automatic scaling by setting minimum and maximum nodes, and the service adds or removes nodes as QPS and token load change. Setting automatic scaling with a minimum of two nodes supports a high availability service level agreement (SLA).

2.5Pairwise Model Quality Evaluation with Autoraters

Pairwise model quality evaluation with autoraters compares a tuned candidate model against a reference model using a judge model aligned to human preferences. The Gen AI evaluation service runs three patterns: tuned first-party model versus reference first-party model with inference running on both models simultaneously, tuned third-party model versus reference third-party model by supplying model responses directly and skipping inference, and alignment checking by supplying both outputs plus a human-preference column. All supported tasks are benchmarked with human-rater data, and supplying human-preference data produces agreement statistics including autorater win rates, human win rates, true positives (TP), false positives (FP), true negatives (TN), false negatives (FN), accuracy, precision, recall, F1, and Cohen's Kappa. Continuous safety evaluation (opens in a new tab) extends the same discipline to content safety, brand safety, relevance, bias and fairness, truthfulness, and robustness to adversarial attacks.

Judge model configuration controls bias, consistency, and domain alignment for these comparisons. AutoraterConfig with flip_enabled=True flips baseline and candidate responses on half the judge calls to reduce position bias in PairwiseMetric evaluations. Multi-sampling calls the judge model multiple times per request to reduce randomness, with sampling_count adjustable from 1 to 32 and a default of 4 that balances consistency against higher request latency. A tuned judge model improves domain-specific judgment by tuning a Gemini base model such as gemini-2.0-flash with training and validation datasets through tune_autorater, or by setting autorater_model to an existing tuned endpoint, while system instructions such as an expert-evaluator role carry higher weight than the evaluation prompt itself.

Selecting and configuring models based on advanced use case parameters is the process of matching model capabilities to task requirements and deployment constraints for low-code AI solutions. The strategy weighs whether a generalized pre-trained model is sufficient or domain-specific tuning is required, and whether the model must be grounded with enterprise data for retrieval-augmented generation (RAG) workflows. Deployment choices for compute, scaling, networking, and observability then determine whether the selected model meets latency, throughput, and availability targets.

3.1Pre-Trained Versus Tunable Models for Specialized Tasks

Pre-trained versus tunable model selection decides whether to deploy a ready-to-use model or train a model on domain data. Pre-trained models perform a specific objective, are trained on generalized data, and are ready to use without additional training. User-trained models require identifying and providing sample data and then training models suited to a unique use case using Vertex AI (opens in a new tab), which offers AutoML trained models and custom-trained models.

Choose a pre-trained model when generalized accuracy is sufficient and time to value matters, and choose a user-trained model when labels, objects, or language patterns are unique to the enterprise. Gemini Enterprise Agent Platform Vision supports Vertex AI AutoML vision models created in October 2022 or later, with compatibility not guaranteed for models trained before that date. In an app graph, data flows from the input source node to the AI model node and then to downstream streams, other models, or a media warehouse. A tuned model can be validated against a reference model from Vertex Model Registry by running inference on both models simultaneously or by supplying stored responses directly for comparison.

Adding a pre-trained Vision model to a low-code application (opens in a new tab) follows an ordered graph construction:

1. Open application: Open the Applications tab and select View app for the target application.
2. Select model: Select the model from the AI models list, which exposes its individual configuration settings.
3. Connect input: Select the source node and add an output connection.
4. Attach model node: Select the newly created AI model node from the list of available nodes to complete the flow from stream to model.

3.2Grounding Models with Enterprise Data for RAG Workflows

Grounding with enterprise data for RAG workflows connects a generative model to retrieved enterprise documents so responses reflect current, domain-specific facts. RAG Engine on Gemini Enterprise Agent Platform uses Gemini Enterprise Agent Platform Vector Search 2.0 as a storage and retrieval system that stores vector representations of documents and retrieves relevant documents based on semantic similarity to a query. RAG Engine manages the Vector Search 2.0 collections for the corpus while leaving those collections visible in the project for transparency.

RAG Engine corpora backed by Vector Search 2.0 are available only in us-central1. The service supports Virtual Private Cloud Service Controls (VPC-SC) security controls and customer-managed encryption keys (CMEK), while data residency and Access Transparency (AXT) controls are not supported. Vector databases for RAG are highly available and multi-zonal or multi-regional, which keeps retrieval available when a zone fails.

Backend selection controls who manages infrastructure, what is visible, and whether CMEK is available:

3.3Serving and Scaling Configuration for Selected Models

Serving and scaling configuration for selected models determines how the selected model receives inputs, what hardware runs it, and how it handles load. Batch prediction scores data at a regular cadence for offline use, while online prediction scores data in near real time for live applications by passing input data to a cloud-hosted model and returning inferences per instance. Clients calling an online endpoint first fetch feature inputs from a database service or from the Feature Store online serving API, then call the prediction endpoint with those inputs, while batch prediction fetches from a data lake or the Feature Store batch serving API.

Latency, availability, and cost depend on network placement, hardware choice, and autoscaling policy. Agent Platform private endpoints lower latency for peer-to-peer requests when the calling application and the serving binary sit in the same local network, because traffic stays inside Virtual Private Cloud (VPC) and avoids internet routing overhead. Deployments select CPU VM types or GPU types appropriate to the model, and custom prediction routines can customize request and response handling to and from the model server. For online prediction, turn on automatic scaling with minimum and maximum nodes, with a minimum of two nodes to support a high availability SLA.

Proactive resource management and traffic distribution keep specialized accelerators available under fluctuating demand. The reliability practices (opens in a new tab) for this configuration include:

• Committed use discounts (CUDs): Use for VMs and accelerators with stable 24x7 inference or fixed, predictable training capacity.
• Spot VMs: Use for fault-tolerant evaluation and experimentation on Google Kubernetes Engine (GKE) nodes and Compute Engine VMs to reduce cost, accepting that Spot VMs can be preempted.
• Dynamic Workload Scheduler (DWS): Use to improve obtainability of high-demand accelerators, with Flex-Start mode for complex batch training up to seven days and Calendar mode to reserve specific GPUs such as H100 and H200 and TPUs such as Trillium for up to three months.
• vLLM engine: Use on GKE to dynamically use TPUs and GPUs and absorb fluctuating inference capacity and performance needs.
• Cluster Director and Ray: Use Cluster Director to deploy accelerator groups with colocation and scheduling for multi-GPU training, and use Ray on Gemini Enterprise Agent Platform to let applications request resources for training and serving without direct management of VMs and containers.

Plan capacity from historical telemetry such as GPU or TPU utilization and throughput from Cloud Monitoring and logs in Cloud Logging (opens in a new tab), analyzed in BigQuery or Looker Studio to forecast accelerator demand. Validate estimates with load testing using tools such as Apache JMeter or LoadView, monitor latency, throughput, errors, and accelerator utilization under stress, and for generative AI serving test under high concurrent load to find where accelerator availability limits performance. Use the model observability dashboard for model QPS, token throughput, and first token latencies, set proactive alerts for agents, and place models behind GKE Inference Gateway for model-aware load balancing across instances. Continuous safety monitoring (opens in a new tab) covers content safety, relevance, bias, truthfulness, and robustness to adversarial attacks.

Key Facts

• The vLLM container supports text-generation and image-text-to-text models, so models outside those tasks cannot use the default one-click deployment path in Model Garden.
• Model-based evaluation metrics can run without ground truth while computation-based metrics usually require ground truth, and pairwise comparison requires Gemini as the judge model.
• Vertex AI online prediction requires a minimum of two autoscaling nodes to support a high availability SLA.
• RAG Engine corpora backed by Vector Search 2.0 are available only in us-central1, and only the RagManagedDb backend backed by Spanner supports customer-managed encryption keys (CMEK).
• Autorater pairwise evaluation uses flip_enabled=True to reduce position bias and multi-sampling with sampling_count from 1 to 32 and a default of 4 to balance consistency against latency.