Prepare and test your skills
Prepare and test your skills
Supervised fine-tuning and parameter-efficient adaptation customize foundation models for exact task behavior without relying on long prompts. Supervised fine-tuning (SFT) teaches a Gemini (opens in a new tab) foundation model a new skill by training on hundreds of labeled input-output examples. In Vertex AI (opens in a new tab), Gemini SFT is implemented as parameter-efficient tuning, where base model parameters stay frozen and only small adapter weights are learned. The learned adapters are combined with the original model for serving, producing a new tuned model plus a serving endpoint that encodes the task behavior.
Supervised fine-tuning is the correct choice when the expected output can be clearly defined, is not overly complex, and deviates from general patterns that prompting alone cannot reliably produce. SFT adjusts model weights to minimize the difference between predictions and actual labels, which makes it effective for domain-specific language that differs significantly from pretraining data. Starting with prompt design is recommended for rapid prototyping, then moving to fine-tuning when 100 or more high-quality examples are available and recurrent errors persist. Choose tuning over prompting also when prompts produce inconsistent results, the task is too complex to articulate, intuitions are difficult to formalize, or removing few-shot examples is needed to reduce context length, latency, and inference cost.
SFT is particularly effective for tasks where format, brevity, or behavior must be exact:
business instead of a verbose explanation.#Person1 and #Person2 and removing personally identifiable information (PII).Last Glacial Maximum for what LGM stands for.Parameter-efficient tuning and full fine-tuning differ in how many weights change and how much compute they require. Parameter-efficient tuning, also called adapter tuning, updates only a relatively small subset of parameters through small adapter layers that work with the frozen foundation model. Only the adapter weights are updated during training, which lowers compute, speeds adaptation, supports smaller datasets, and enables multi-task learning without extensive retraining. Full fine-tuning updates all model parameters, which can suit highly complex tasks but demands much higher compute for both tuning and serving.
| Approach | Parameters updated | Resource and use profile |
|---|---|---|
| Parameter-efficient tuning | Adapter weights only; base model frozen | Lower training cost, faster adaptation, smaller datasets; choose for Gemini SFT and specialized domain tasks |
| Full fine-tuning | All model parameters | Higher tuning and serving cost; choose only when task complexity requires whole-model adaptation |
Related customization options use different mechanisms. Reinforcement Learning from Human Feedback (RLHF) tuning and preference tuning build on SFT to learn subjective human preferences for complex outputs such as question answering and creative generation. Distillation trains a smaller student model to mimic a larger teacher model, while grounding connects outputs to verifiable sources during inference without changing weights.
Dataset preparation for Gemini SFT starts with a labeled JSONL file stored in Cloud Storage (opens in a new tab), with separate requirements for text, image, document, audio, video, and function-calling use cases. Each training example demonstrates the desired output for a given input during inference, and the data distribution, format, and context should match what the model will see in production. Quality matters more than quantity, so analysis of where the base model fails should guide data curation before adding more examples. Validation data is optional but required for validation metrics that reveal generalization.
Documented limits and defaults define the job envelope. Maximum input and output training tokens are 131,072, while serving tokens remain the same as the base Gemini model. Maximum training file size is 1 GB for JSONL, with up to 10M text-only examples or 300K multimodal examples. Validation data is limited to a maximum of 5000 examples or 30% of the number of training examples when there are more than 1000 validation examples.
A supervised tuning job in Gemini Enterprise Agent Platform runs a pipeline that reads data from Cloud Storage, learns adapter parameters, and deploys the tuned model and endpoint in the selected tuning job region. Supported base models include Gemini 3.5 Flash, Gemini 3.1 Flash-Lite, Gemini 2.5 Pro, Gemini 2.5 Flash, and Gemini 2.5 Flash-Lite. User data such as the transformed dataset and tuned model is stored in the tuning job region, while compute may be transparently offloaded to other US or EU regions for accelerator availability. Training tokens billed are tokens in the training dataset multiplied by number of epochs, and quota is a global concurrent tuning jobs limit shared across regions and models with a default of at least one job.
The console flow in Agent Platform Studio follows an ordered sequence:
The same job can be created with the Google Gen AI SDK, REST tuningJobs.create method, or Colab Enterprise. The SDK uses TuningDataset with a gcs_uri for training and validation, CreateTuningJobConfig for display name and datasets, and polls while state is JOB_STATE_PENDING or JOB_STATE_RUNNING. A completed job returns tuned_model.model, tuned_model.endpoint, an experiment context, and per-checkpoint details with checkpoint_id, epoch, step, and endpoint. Supported adapter size values are 1, 2, 4, 8, and 16, with some models supporting 1, 2, 4, and 8.
Exam tip: Do not apply controlled generation at inference time to a supervised-tuned Gemini model to get structured output, because tuning did not use controlled generation and SFT already customizes structured output.
Evaluation of a tuned Gemini model combines built-in tuning and validation metrics with optional automated evaluation through the Gen AI evaluation service. Tuning metrics track learning progress during the job, while validation metrics computed on the held-out Cloud Storage validation set reveal whether the adapter is generalizing or memorizing the training set. Checkpoints let progress be saved, checkpoint performance be compared, and the best-performing checkpoint be selected instead of only the final state. Integrated evaluation is configured with evaluationConfig and runs automatically after tuning completes, currently in Preview in us-central1 through the Gen AI SDK and REST API for gemini-2.5-pro, gemini-2.5-flash, and gemini-2.5-flash-lite.
The evaluation configuration specifies metrics such as FLUENCY with a prompt template and an OutputConfig with a GcsDestination prefix for results, and evaluations are charged as batch prediction jobs. Continuous tuning can then continue from an already tuned model or checkpoint with more epochs or examples, which is more practical than full retraining for foundation models. Overfitting and catastrophic forgetting are controlled by keeping epochs, adapter size, and learning rate multiplier small enough to learn the task without overwriting general capability. If validation metrics degrade while training metrics improve, reduce epochs, reduce adapter size, lower the learning rate multiplier, improve validation coverage, or select an earlier checkpoint. For dynamic applications such as chatbots that drift with human expectations, plan more frequent continuous tuning with human feedback data, using GPUs (opens in a new tab) for parallel intensive work or TPUs for large-matrix deep-learning operations.
Deploying tuned and Model-as-a-Service endpoints means placing foundation models behind scalable inference APIs that combine model serving, retrieval grounding, and safety controls. In Google Cloud, Vertex AI (opens in a new tab) serves first-party models such as Gemini, Imagen, Codey, and text-embedding as serverless Model-as-a-Service (MaaS) APIs, while Model Garden (opens in a new tab) provides deployable open-source models such as Llama 2 70B to dedicated serving infrastructure on Vertex AI, GKE, Cloud Run, or Compute Engine. A production endpoint therefore connects three planes: a model endpoint for generation, a retrieval plane for grounding, and a control plane for security, safety filtering, and quality evaluation.
Model Garden is the Vertex AI catalog for discovering Google and open-source foundation models and deploying them to managed endpoints for online inference. Model selection follows modality: Gemini (Vertex AI) handles text and multimodal text-plus-image-or-video prompts including flagship capability and Google Search grounding, Imagen (Vertex AI) generates images, Codey (Vertex AI) generates code, and text-embedding models generate vectors for search, classification, or clustering. Once selected, a tuned or third-party model is deployed as a Vertex AI endpoint for autoscaled serving, or as a containerized service on GKE for open-source toolchains with Ray, Hugging Face, and LangChain, on Cloud Run for serverless frontends and retrieval services, or on Compute Engine with GPUs for manual control.
The serving path isolates frontends from inference logic to support scale and multiple clients. Traffic flows from the user to a Cloud Load Balancing frontend, then from the load balancer backend to a Cloud Run frontend service, then from the frontend service to a retrieval service, then from the retrieval service to Vertex AI for embedding and generation. The retrieval service depends on Secret Manager for API keys and database credentials and on Cloud SQL or AlloyDB for vector lookup, so credentials never live in application code and retrieval scales independently from generation. Choose serverless MaaS when the priority is zero infrastructure management for Gemini, Imagen, or embeddings, and choose a dedicated Vertex AI, GKE, or Compute Engine deployment when the workload needs custom containers, accelerator sizing, or VPC-attached open-source serving.
Grounding is the practice of augmenting a prompt with retrieved real-time or proprietary data before generation, also called retrieval-augmented generation (RAG). The RAG workflow depends on an embedding model, a vector store, and a generator: the retrieval service converts the user request to an embedding, searches for similar vectors, then sends the retrieved passages plus the original prompt to Vertex AI to create a grounded response. Gemini models can also evaluate whether internal knowledge is sufficient or whether grounding with Google Search is required, so grounding can use owned indexes, live web data, or both.
Vector storage selection depends on data shape, latency, and management overhead. Vertex AI Search is a fully managed search engine that supports most data sources without exposing underlying embeddings, Vertex AI RAG Engine is a managed orchestrator with a LlamaIndex-like interface, Vertex AI Vector Search is a specialized low-latency index for very large-scale serving, AlloyDB for PostgreSQL and Cloud SQL with pgvector serve OLTP workloads that already live in Postgres and need low-latency SQL plus in-database AI, and BigQuery serves OLAP analytical datasets with batch processing and human or script SQL access.
| Option | Management model | Choose when |
|---|---|---|
| Vertex AI Search | Fully managed search engine | The workload needs optimized retrieval over many sources without managing embeddings |
| Vertex AI Vector Search | Fully managed specialized vector index | The workload needs low-latency, large-scale vector serving |
| AlloyDB for PostgreSQL and Cloud SQL | Managed Postgres with pgvector | Data is OLTP, already in SQL, and needs low-latency programmatic access from the database |
| BigQuery | Managed analytical warehouse | Data is large-scale OLAP and needs batch SQL analytics over tables |
Exam tip: Choose Vertex AI Search to avoid managing embeddings, Vertex AI Vector Search for large-scale low-latency vector serving, and Cloud SQL or AlloyDB when grounding data already lives in Postgres and needs transactional access.
Gemini Enterprise is the fully managed platform for running enterprise RAG agents, and its architecture separates ingestion from serving so new data becomes retrievable with minimal delay. The data ingestion subsystem moves data through an ordered pipeline where each stage triggers the next through storage events and messaging. Enrichment during processing allows business logic such as tagging product line, author, location, or document type, so later search can filter on metadata for precise, context-aware answers.
The ingestion pipeline for Gemini Enterprise runs in this order:
The serving subsystem handles the query-response flow between users and the grounded model:
This split keeps frontend clients thin, centralizes filtering and orchestration in the backend, and lets the agent orchestrate retrieval and generation under Responsible AI safety filters.
Secure RAG endpoints isolate identity, encryption, network, and data-residency controls across every hop from frontend to model to database. The retrieval service authenticates to Vertex AI and Cloud SQL (opens in a new tab) with credentials pulled at runtime from Secret Manager, so keys rotate centrally and never ship with containers. AlloyDB for PostgreSQL encrypts stored data by default with Google-owned keys and supports customer-managed encryption keys (CMEK) when the organization must control keys, accepts only SSL connections by default, and supports the Auth Proxy connector and language connectors that provide Identity and Access Management (IAM)-based authorization over a TLS 1.3 connection with a 256-bit AES cipher. Network exfiltration risk is reduced with a VPC Service Controls perimeter around AlloyDB, while Cloud Storage supports IAM over access control lists (ACLs), Google-managed encryption by default, and CMEK or customer-supplied encryption keys (CSEKs) for sensitive ingestion buckets.
Compliance extends to residency, governance, and output safety in the same serving chain. AlloyDB and BigQuery store or replicate data only within specified regions, and BigQuery adds data governance for access control, sensitive-data protection, and consistency for offline analytics. After generation and before delivery to the frontend, the application screens responses with responsible AI filters, then loads prompts, responses, and log data to BigQuery for audit and analysis. Because screening sits between generation and the frontend, unsafe output is blocked before it reaches users while every decision remains traceable in logs, Cloud Logging, and Cloud Monitoring.
The quality evaluation subsystem measures factual accuracy and relevance of serving responses so tuned models and grounding indexes can be improved over time. Evaluation runs asynchronously from serving through an ordered flow that separates live latency from offline quality work. Serving stays low-latency on Vertex AI, Vector Search, and Cloud Run (opens in a new tab), while BigQuery, Cloud Logging, and Cloud Monitoring provide the evidence for retuning models, adjusting chunking and metadata, or switching retrieval options.
The evaluation loop for serving quality runs in this order:
Use this pattern for support chatbots grounded in technical docs, HR assistants grounded in policy stores, legal researchers grounded in contracts and case law, semantic product search grounded in catalogs, personalized recommendations grounded in behavior, and clinical assistants grounded in records and knowledge bases such as PubMed.
Multimodal solutions on Google Cloud combine managed foundation models for each modality with open models curated in Model Garden. Gemini provides multimodal reasoning and function calling, Imagen provides image generation and editing, Veo (opens in a new tab) provides text-to-video and image-to-video synthesis, and Model Garden provides Google, open-source, and third-party models that can be consumed as a managed API or deployed to dedicated infrastructure.
Foundation-model selection starts from input and output modalities and operational constraints for latency, throughput, cost, and data residency. Veo is the choice when the output is video generated from a text prompt, optionally anchored by first-frame and last-frame images. Multimodal embeddings are the choice when text, image, and video must share one retrieval space for search, recommendation, or retrieval-augmented generation (RAG). Open models from Model Garden are the choice when a business needs domain specialization, open weights, or deployment inside its own project boundary.
The documentation supports three multimodal building blocks for this decision:
imageEmbedding, textEmbedding, and segmented videoEmbeddings with startOffsetSec and endOffsetSec for cross-modal similarity.Choose a managed API when speed of integration matters most, and choose a Model Garden deployment when control over accelerator, serving stack, scaling, and data location matters most.
Veo is a long-running generative model that synthesizes video from a text prompt and optional image conditioning. The request flows from the client application to the Veo endpoint, the service starts an operation, and the generated video flows to the Cloud Storage bucket specified by output_gcs_uri or is returned inline as video bytes. Because generation takes minutes, the client must poll the operation until operation.done is true and then read generated_videos[0].video.uri. The first-frame image defines initial composition and the last-frame image defines final composition, so text, start image, and end image together constrain the full trajectory of the clip.
A Veo image-to-video request follows an ordered workflow:
model as veo-3.1-generate-001 or veo-3.1-fast-generate-001, a prompt describing motion and scene, and an image for the first frame stored in Cloud Storage as image/jpeg or image/png.last_frame to force the clip to end on a specific image, plus aspect_ratio such as 16:9, output_gcs_uri such as gs://video-bucket/output/, sampleCount from 1-4, and seed for reproducibility.client.models.generate_videos or endpoints.predict, which returns an operation handle rather than immediate video.client.operations.get(operation) until done, then read the output URI from the operation response.Veo 3 models accept durations of 4, 6, or 8 seconds with a default of 8, while Veo 2 models accept 5-8 seconds with a default of 8. The first-frame and last-frame conditioning (opens in a new tab) controls continuity, which is why those model versions are required when both start and end images must constrain the clip.
Exam tip: Use veo-3.1-generate-001 or veo-3.1-fast-generate-001 when first-frame or last-frame conditioning is required, and remember Veo 3 durations are only 4, 6, or 8 seconds with sampleCount 1-4.
Model Garden is the Vertex AI registry for discovering Google, open-source, and partner models and serving them as Model as a Service (MaaS) or as dedicated deployments. MaaS (opens in a new tab) exposes the model behind a serverless API with no accelerator management, while a Model Garden deployment (opens in a new tab) provisions the model on Vertex AI endpoints inside the customer's project with chosen GPUs or TPUs. A custom container deployment packages the same weights with an optimized serving stack such as vLLM for full control over throughput tuning and versioning.
| Option | Operational unit | Choose when |
|---|---|---|
| Model as a Service (MaaS) | Shared managed API endpoint | The priority is lowest operational overhead and elastic scaling without managing accelerators |
| Deploy open models from Model Garden | Dedicated Vertex AI endpoint in customer project | The priority is data residency, VPC control, custom scaling, and accelerator selection including TPU |
| Deploy with custom vLLM container | Self-managed serving container on Vertex AI | The priority is custom inference tuning, newer runtimes, or framework-specific optimizations |
Traffic in all three options flows from the application to a Vertex AI endpoint, but ownership of the compute boundary differs. MaaS keeps the compute pool on the provider side, while Model Garden and custom-container deployments keep model processes, scaling policy, and output storage on the customer side. That boundary determines who controls cold-start latency, sustained throughput, accelerator cost, and where prompts and artifacts reside.
The serving stack determines how an open model uses accelerators to meet latency and throughput targets. vLLM, SGLang, and TensorRT-LLM provide high-throughput LLM serving with continuous batching and optimized attention kernels, Text Embeddings Inference (TEI) provides high-throughput embedding serving, PyTorch Inference provides general Hugging Face model serving, and Hex-LLM provides TPU-optimized serving for Llama-class models. The model artifact flows from Model Garden to the selected runtime, the runtime loads weights onto accelerators, and prediction traffic then flows from the Vertex AI endpoint to that runtime, so changing the runtime changes batching behavior and accelerator utilization without changing model weights.
The documentation maps stacks to representative models in prebuilt containers (opens in a new tab):
Choose TEI for embedding throughput, vLLM or SGLang or TensorRT-LLM for high-concurrency LLM decoding, and Hex-LLM when the deployment target is TPU.
Supervised fine-tuning (SFT) is the correct choice when the expected output can be clearly defined, is not overly complex, and deviates from general patterns that prompting alone cannot reliably produce. Starting with prompt design is recommended for rapid prototyping, then moving to fine-tuning when 100 or more high-quality examples are available and recurrent errors persist, or when prompts produce inconsistent results and removing few-shot examples is needed to reduce context length, latency, and inference cost. SFT adjusts model weights to minimize the difference between predictions and actual labels, which makes it effective for domain-specific language that differs significantly from pretraining data.
Parameter-efficient tuning updates only small adapter weights while the base model parameters stay frozen, while full fine-tuning updates all model parameters. Parameter-efficient tuning lowers compute, speeds adaptation, supports smaller datasets, and enables multi-task learning without extensive retraining, while full fine-tuning demands much higher compute for both tuning and serving. Choose parameter-efficient tuning for Gemini supervised fine-tuning and specialized domain tasks and full fine-tuning only when task complexity requires whole-model adaptation.
Veo synthesizes video through a long-running operation where the client starts generation and polls the operation until done is true and then reads the generated video output URI from the operation response. A Veo image-to-video request supplies a model such as veo-3.1-generate-001, a prompt describing motion and scene, and a first-frame image in Cloud Storage, with optional last-frame image, aspect ratio, output Cloud Storage URI, sampleCount from 1 to 4, and seed for reproducibility. Veo 3 models accept durations of 4, 6, or 8 seconds with a default of 8, where the first-frame image defines initial composition and the last-frame image defines final composition.
Grounding augments a prompt with retrieved data where the retrieval service converts the request to an embedding, searches for similar vectors, then sends retrieved passages plus the original prompt to Vertex AI to create a grounded response. Choose Vertex AI Search when the workload needs optimized retrieval over many sources without managing embeddings, Vertex AI Vector Search when the workload needs low-latency large-scale vector serving, AlloyDB for PostgreSQL and Cloud SQL with pgvector when data is OLTP, already in SQL, and needs low-latency programmatic access, and BigQuery when data is large-scale OLAP and needs batch SQL analytics. Gemini models can also evaluate whether internal knowledge is sufficient or whether grounding with Google Search is required, so grounding can use owned indexes, live web data, or both.
Last updated