Fine-Tuning Foundation Models for Vertex AI

Assessing performance, deploying tuned adapters, and integrating with agents is the production phase where a fine-tuned checkpoint becomes a governed, served capability. In Vertex AI Model Garden (opens in a new tab), a base foundation model is selected and tuned, while in Gemini Enterprise Agent Platform the resulting artifact is evaluated, registered, deployed to an inference endpoint, and consumed by an agent. The tuned adapter does not replace the base model; it attaches task-specific behavior to the frozen base weights while endpoint infrastructure and agent orchestration remain managed services.

Data flows from BigQuery or Cloud Storage into training, then into automated evaluation, then into Model Registry and Inference. Vertex AI Pipelines (opens in a new tab) automates this lifecycle from data preparation to training, evaluation, and deployment. Cloud Build connects to Pipelines for continuous training and testing, so new data or prompts can trigger retraining, bulk evaluation, stress testing, and a controlled release to production.

1.1Evaluate Tuned Model Quality with Automated Gen AI Metrics

Automated evaluation of a tuned generative model is the use of managed evaluation primitives to score generated outputs against quality metrics and safety guidelines without manual review of every response. The Gen AI evaluation service evaluates any generative model or application and benchmarks results against defined judgment criteria, including specialized checks for prompt engineering, retrieval-augmented generation (RAG), and AI agent optimization. The Agent Platform evaluation service provides ready-to-use primitives to track model performance as part of the ML lifecycle, so loss, accuracy, text quality, image fidelity, code functionality, and metrics like perplexity are logged consistently across runs.

Evaluation data flows from BigQuery or Cloud Storage into the evaluation service. BigQuery lets teams use SQL to run inference tasks like natural language processing and machine translation for large-scale analysis, while Cloud Storage provides cost-efficient storage for large evaluation datasets. Agent Platform Pipelines then automates evaluation of generated outputs, including response filtering, data augmentation for prompt diversity, and human-in-the-loop review for nuanced aspects like creativity and safety. For generative systems, testing must also monitor latency, error rate, and resource utilization through Cloud Monitoring (opens in a new tab), because a tuned adapter that improves quality but violates latency or safety thresholds cannot be promoted.

1.2Compare Candidates with Side-by-Side Analysis

Side-by-side comparative analysis is pairwise evaluation that directly compares outputs from two model versions on the same prompts to decide which behavior to ship. The Agent Platform automatic side-by-side (AutoSxS) tool supports pairwise model-based evaluation, where a judge model scores which response better meets the criteria. This comparison complements aggregate metrics like accuracy, precision, and recall on test datasets and subset analysis across data slices, model versions, and architectures.

The standard evaluation pipeline combines lifecycle tracking and pairwise comparison to make promotion decisions:

1.3Deploy Tuned Adapters as Versioned Production Endpoints

Deployment of a tuned adapter is the registration of the tuned artifact as a versioned model and its release to a managed inference endpoint with monitoring and rollback. The tuned artifact flows from the tuning job to Model Registry, which stores, organizes, and versions model binaries and metadata, then to Inference, which serves predictions on serverless architecture. Because Model Registry preserves lineage to parameters, metrics, and dataset artifacts through Vertex ML Metadata and Experiments, any endpoint can be traced back to the base Model Garden model, training data, and evaluation scores that justified its release.

A safe production release follows an ordered sequence:

1. Register version: Store the tuned adapter and its metadata as a new version in Model Registry, preserving linkage to the base model and evaluation results.
2. Validate in CI/CD: Run Cloud Build test suites for packaging, dependencies, and infrastructure, plus stress tests with tools like Locust, Grafana k6, or Apache JMeter and bulk evaluations across diverse datasets.
3. Release in controlled stages: Deploy to Inference with a release approach that validates performance on live traffic, detects issues early, and rolls back quickly to the prior version when required.
4. Enable continuous training: Schedule Pipeline runs for automated retraining on new data or prompts, selecting the latest base models from Model Garden so tuned adapters remain current.

After deployment, Model Monitoring watches the endpoint for data drift, performance degradation, and anomalies. For generative systems, Model Monitoring specifically tracks shifts in output quality and safety compliance, so a drift in tone, factuality, or refusal behavior triggers retraining or rollback even when traditional accuracy remains stable.

1.4Integrate Tuned Models into Agent Architectures

Integration with agents is the connection of a deployed tuned endpoint to Vertex AI Agent Builder architectures as the reasoning and generation engine behind tools, retrieval, and dialogue. The agent sends a prompt and context to the tuned endpoint, the endpoint returns a domain-adapted completion, and the agent routes that completion through response filtering, tool calls, and RAG grounding before responding to the user. The tuned adapter therefore depends on the endpoint for serving and the agent depends on the tuned adapter for task-specific behavior, while Feature Store, Vector Search, and BigQuery supply the features and retrieval context the agent passes in.

Agent Platform Pipelines automates the agent-specific steps of prompt engineering, response filtering, and human-in-the-loop evaluation, including evaluation of generated outputs against quality metrics and safety guidelines. The Gen AI evaluation service directly supports AI agent optimization by benchmarking agent responses against judgment criteria, which lets teams iterate on prompts, retrieval configuration, and adapter checkpoints together. To keep agent behavior relevant and diverse, continuous training automatically updates models with new training data or prompts, ensuring the deployed adapter evolves with changing business needs and production monitoring signals.

Model customization on Gemini Enterprise Agent Platform (opens in a new tab) is a spectrum of strategies that trade task complexity, data requirements, latency, and cost against depth of adaptation. The spectrum runs from prompt engineering with a frozen pre-trained model, to grounding with Retrieval-Augmented Generation (RAG) (opens in a new tab), to parameter-efficient fine-tuning (PEFT), to full supervised fine-tuning (SFT). Selection depends on whether the behavior can be described in instructions, retrieved at inference time, or must be encoded into model weights through labeled examples.

2.1Prompt Engineering as the Baseline

Prompt engineering is the practice of crafting instructions, format descriptions, and context to guide a frozen Gemini model without changing its weights. Prompting with pre-trained Gemini models requires minimal setup and is best suited for rapid prototyping and for cases with limited labeled data where a lengthy tuning process cannot be justified. The recommended path is to start with prompting to find the optimal prompt, evaluate where the model makes mistakes, and only then move to tuning to boost performance or fix recurrent errors.

Tuning provides distinct advantages over prompt design once that threshold is crossed. A tuned model delivers deep customization with better performance on specific tasks, alignment to custom syntax and domain-specific semantic rules, and more consistent and reliable results. Because the behavior is encoded in learned parameters rather than repeated instructions, tuning can handle more examples at once and saves inference cost and latency by removing few-shot examples and long instructions from prompts. In short, data flows from many training examples into model weights during tuning, so data does not need to flow through the prompt on every inference request.

2.2Grounding and Retrieval-Augmented Generation Without Retraining

Grounding is an adaptation strategy that lets a foundation model produce current, context-specific answers without retraining the entire model. In this pattern, data flows from external or internal sources such as websites, documents, PDFs, databases, vector databases, or APIs like Google Search into the prompt at inference time. A RAG-based application can rewrite the user query, retrieve relevant chunks or curated few-shot examples from a vector database, and combine that retrieved content with a prompt template and conditioning prompts that set boundaries on generation.

Grounding preserves a different trust and lifecycle boundary than tuning. The proprietary knowledge remains in the retrieval store and prompt context rather than being baked into model weights, while the foundation model itself stays frozen. That separation requires operational work in MLOps pipelines: processing data into embeddings, optimizing chunking strategies, versioning prompt templates, and tracking user queries, retrieved passages, and intermediate chain states end to end. Choose RAG alone when freshness and source separation matter most, and combine RAG with tuning when the model also needs a stable domain style, format, or behavior that retrieval alone cannot enforce.

2.3Supervised Fine-Tuning for Well-Defined Domain Tasks

Supervised fine-tuning (SFT) (opens in a new tab) is a tuning method that teaches a Gemini model a new skill by training it to predict the correct output sequence for each labeled input. A tuning job consumes a training dataset of hundreds of labeled examples where each example demonstrates the desired inference behavior, learns additional parameters that encode that task, and outputs a new model that combines the newly learned parameters with the original model. The process adjusts weights to minimize the difference between predictions and labels, which makes SFT particularly effective when language or content differs significantly from pretraining data.

Supervised fine-tuning of a text model is a good option when the output is not complex and is relatively easy to define with labels. Recommended tasks include:

• Classification and sentiment analysis where the response must be a specific word or phrase rather than a verbose answer
• Entity extraction and extractive question answering where the answer is a substring or specific phrase from the context
• Summarization of content that is not complex, including strict formats such as replacing speaker names with #Person1 and #Person2 and removing personally identifiable information (PII)
• Domain-specific queries and chat persona control, such as forcing a shopkeeper role that refuses out-of-scope requests

For code models, supervised tuning is the only supported tuning option.

2.4Parameter-Efficient Tuning Versus Full Fine-Tuning

Parameter-efficient tuning, also called adapter tuning, is an approach that updates only a relatively small subset of model parameters during tuning. Low-Rank Adaptation (LoRA) (opens in a new tab) is the PEFT mode for open models such as Llama 3.1 that adjusts a subset of parameters through a configurable adapter size, where a larger adapter size lets the model learn more complex tasks but requires a larger dataset and longer training. For Gemini supervised tuning, supported adapter sizes include 1, 2, 4, 8, and 16 on most models, with Gemini 2.5 Pro supporting 1, 2, 4, and 8.

Full fine-tuning updates all parameters of the model, which gives it higher quality potential for highly complex tasks but demands higher computational resources for both tuning and serving. That cost is driven by accelerator requirements: graphics processing units (GPUs) provide parallel processing for intensive workloads, while tensor processing units (TPUs) accelerate large-matrix operations common in deep networks, and both directly affect tuning speed and expense. The tuned artifact boundary also differs: a PEFT job produces adapter parameters that combine with the frozen base at inference, while a full job produces a fully updated weight set that is more expensive to store, serve, and maintain. Choose PEFT when data is limited to hundreds of examples, iteration speed and cost matter, or multiple tasks must share a base; choose full fine-tuning only when task complexity exceeds adapter capacity and the quality gain justifies higher training and serving cost.

2.5Distillation and Preference Tuning for Reasoning and Subjective Alignment

Distillation fine-tuning and preference tuning extend SFT for two specialized gaps: transferring reasoning and aligning to subjective human judgment. Distillation tuning uses a larger teacher model to generate completions that are then used to tune a smaller student model, with the training dataset allowed to contain prompts only because the teacher supplies the labels. Distillation is most effective when the teacher is substantially more capable than the student on the target task, especially for transferring complex multi-step reasoning such as math and quantitative reasoning, scientific and medical question answering that requires step-by-step reasoning, and other thinking or chain-of-thought behaviors.

Preference tuning builds on SFT by tuning with human feedback data, which lets the model learn subjective preferences that are hard to define with specific labels alone. These methods change data dependencies and quota boundaries. A distillation job depends on both the student base model and the teacher model, so the project must hold sufficient quota for the teacher, which for models offered as a service consumes dynamic shared quota on each teacher call. Preference tuning depends on a prior SFT foundation and on curated preference comparisons rather than single correct answers. Choose distillation when a small deployable model must inherit reasoning from a large thinker, and choose preference or reinforcement learning from human feedback (RLHF) when success is defined by human preference, intricate instruction following, or creative quality rather than one exact label.

2.6Comparing Strategies by Task, Data, Latency, and Cost

The comparison of customization strategies shows how prompting, grounding, and tuning differ in what changes, what data they require, and what they cost at inference. Prompting and RAG leave weights frozen and vary only inputs, while PEFT and full SFT change weights and reduce per-request prompt size. Training token cost for Gemini SFT is calculated as tokens in the training dataset multiplied by epoch count, with up to 131,072 input and output tokens per training example, up to 1 GB JSONL files, and up to 10M text-only or 300K multimodal examples, so larger adapters and more epochs directly increase tuning cost.

In practice the strategies compose rather than exclude each other. A team prototypes with prompts (opens in a new tab), adds RAG to keep answers current and grounded, applies SFT with PEFT to lock in format and domain semantics and cut latency, and reserves full tuning, distillation, or preference tuning for the hardest reasoning or alignment gaps. Continuous tuning then extends the chosen tuned model or checkpoint with more epochs or examples as data drifts, which is more practical than full retraining for generative models.

Fine-tuning workflows in Vertex AI Model Garden (opens in a new tab) and Gemini Enterprise Agent Platform adapt a proprietary Gemini foundation model or an open-weight model such as Llama 3.1 to a downstream task. Data flows from JSON Lines (JSONL) datasets in Cloud Storage into a managed tuning job, and the job writes a tuned model, checkpoints, and experiment metadata to a Cloud Storage output directory and deploys a servable endpoint. The job runs in a selected region, and that region is also where the tuned model is uploaded.

3.1Prepare and Validate Supervised Datasets in Cloud Storage

Supervised dataset preparation stores training and optional validation examples as JSONL files in Cloud Storage and points the tuning job at their gs:// URIs. Each line in the file contains one tuning example, and best results use at least 100 to 500 high-quality, well-labeled examples that reflect the prompt distribution, format, and context the model will see in production. The training dataset teaches the desired behavior, while the validation dataset enables validation metrics during training when model validation is enabled.

Text-only open-model datasets use supported JSONL shapes that the tuning service parses into prompts and ideal completions:

• Prompt-completion objects with prompt and completion fields for single-turn instruction examples
• Turn-based chat objects with a messages array of user and assistant roles for multi-turn dialogue
• GenerateContent objects with systemInstruction and contents parts, including file_data references for multimodal inputs

For distillation tuning, the training dataset can contain prompts only because the teacher model generates the completions, but the validation dataset must contain both prompts and ground-truth completions because it evaluates the student on held-out examples. Multimodal examples can reference images by Cloud Storage URI such as gs://your-gcs-bucket/your-image.jpeg or by base64 data, using JPEG, PNG, and WEBP formats. If images live in a different bucket than the JSONL files, both buckets must grant the Storage Object User role to service-PROJECT_NUMBER@gcp-sa-vertex-moss-ft.iam.gserviceaccount.com and service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com.

3.2Configure Training Hyperparameters for Foundation Models

Training hyperparameters (opens in a new tab) control how many passes the model makes over the data, how many parameters can change, and how large each optimization step is. Submit the first job with recommended defaults, which Agent Platform benchmarks and auto-adjusts to dataset size, then adjust only to fix recurrent errors after evaluating where the model fails. Quality of labeled data matters more than quantity.

The core supervised hyperparameters are epochs, adapter size, and learning rate multiplier. Epochs define the number of complete passes over the training dataset, and the service sets a recommended default from training dataset size. Adapter size controls the number of trainable adapter parameters; a larger adapter can learn more complex tasks but needs a larger dataset and longer training time. Learning rate multiplier scales the recommended step size at each iteration, with a default value of 1. An additional Export last checkpoint only option disables intermediate checkpoints and keeps only the latest checkpoint.

3.3Orchestrate Fine-Tuning Jobs with Model Garden and Agent Platform Tools

A fine-tuning job binds a base model, Cloud Storage datasets, hyperparameters, and an output location into a managed pipeline that produces a tuned model resource, an endpoint, and an experiment record. Supported Gemini bases for supervised tuning (opens in a new tab) include Gemini 3.5 Flash, Gemini 3.1 Flash-Lite, Gemini 2.5 Pro, Gemini 2.5 Flash, and Gemini 2.5 Flash-Lite; open-model tuning can start from a supported base such as meta/llama3_1@llama-3.1-8b or from a same-architecture custom checkpoint or previously tuned output via custom_base_model for continuous tuning.

A console-based supervised job follows an ordered workflow:

1. Define the model: Open Agent Platform Studio, click Create tuned model, then set a tuned model name up to 128 characters, select the base model, and select the pipeline region.
2. Set tuning configuration: Enter number of epochs, adapter size, and learning rate multiplier, and optionally enable Export last checkpoint only.
3. Attach datasets: Choose Upload file to Cloud Storage or Existing file on Cloud Storage for the training JSONL file, select the destination or source bucket, and optionally enable model validation and supply the validation dataset Cloud Storage path.
4. Start and monitor: Click Start Tuning, then track the job under Tuned Models or the Managed tuning tab until Status shows Succeeded and review the Monitor, Dataset, and Details tabs.
5. Consume outputs: Record the tuned model name such as projects/123456789012/locations/us-central1/models/1234567890@1, the endpoint name, the experiment context, and any per-checkpoint endpoints.

The same job can be orchestrated programmatically. The Google Gen AI SDK calls client.tunings.tune with base_model, a TuningDataset for gcs_uri, and a CreateTuningJobConfig carrying tuned_model_display_name, validation_dataset, and an optional preview evaluationConfig for automatic Gen AI evaluation in us-central1. The Vertex AI SDK for Python (opens in a new tab) calls sft.train with a SourceModel, tuning_mode of FULL or PEFT_ADAPTER, epochs, train_dataset, validation_dataset, and output_uri, then polls until has_ended is true and refreshes the job. The REST method tuningJobs.create accepts baseModel, supervisedTuningSpec with training_dataset_uri, validation_dataset_uri, and hyperParameters containing epochCount, adapterSize, and learningRateMultiplier, plus optional tunedModelDisplayName, encryptionSpec.kmsKeyName, serviceAccount, and metric specs such as pointwise, pairwise, exact match, BLEU, and ROUGE. Colab Enterprise adds the same parameters through the Tuning side panel and exposes a View tuning job and Test path to chat with the result.

3.4Manage Checkpoints, Artifacts, Validation, and Deployment

Tuned model artifacts preserve training progress and the servable weights so the best version can be selected and deployed. Gemini tuning supports tuning checkpoints for saving progress, comparing performance, and choosing the best checkpoint, plus continuous tuning to resume an already tuned model or checkpoint with more epochs or examples. Open-model output in the Cloud Storage output directory includes distillation_labelled_dataset.jsonl for distillation jobs and postprocess/node-0/checkpoints/ containing a final/ directory and intermediate checkpoint-M through checkpoint-N directories with sharded model-00001-of-000xx.safetensors files.

Checkpoint retention follows strict limits that determine how many candidates remain available. A maximum of 10 checkpoints are stored, and if the number of epochs is less than 10, exactly that many checkpoints are stored with one per epoch. Intermediate checkpoints are ordered but not always consecutively numbered, so a run may retain checkpoints 1, 3, 5, and 10 rather than 1 through 4. Tuning and validation metrics, automatic evaluation outputs to a Cloud Storage prefix, and experiment metadata then guide whether to deploy the final model or an earlier checkpoint. Deployment moves weights from Cloud Storage to an Agent Platform endpoint (opens in a new tab) for online inference.

Deployment moves weights from Cloud Storage to an Agent Platform endpoint for online inference. The console path uses Model Garden and Deploy model with custom weights, while the SDK path builds a CustomModel from gs://{STORAGE-URI}/postprocess/node-0/checkpoints/final and calls deploy on a GPU machine such as g2-standard-12 with one NVIDIA_L4 accelerator. Deployed custom models serve through predict, while managed open models use chat.completions. Quota limits concurrent jobs globally through Global concurrent managed OSS model fine-tuning jobs per project, with a default of at least one job, and distillation jobs also consume dynamic shared quota for the teacher model.

Key Facts

• A tuned adapter attaches task-specific behavior to a frozen base model and flows from the tuning job to Model Registry for versioning and then to Inference for serving, with lineage preserved through Vertex ML Metadata and Experiments.
• The Gen AI evaluation service scores one tuned model against quality and safety criteria, while the AutoSxS tool performs pairwise comparison of two models on the same prompts to choose a winner.
• Prompt engineering and RAG leave model weights frozen, while PEFT/LoRA updates a small adapter subset and full supervised fine-tuning updates all parameters at higher training and serving cost.
• Supervised tuning requires 100 to 500 high-quality labeled examples in JSONL files in Cloud Storage, with distillation training data allowed to be prompts only but validation data requiring ground-truth completions.
• A maximum of 10 tuning checkpoints are stored with one per epoch when epochs are fewer than 10, and Model Monitoring watches deployed endpoints for data drift and shifts in output quality and safety compliance.