Professional Machine Learning Engineer

Select and Architect Multimodal Solutions Using Foundation Models and Model Garden

Video Generation with Veo

Veo is Google's foundation model family designed to generate high-definition video clips from text prompts, starting images, or reference assets. The family includes model variants such as veo-2.0-generate-001, veo-3.1-generate-001, and veo-3.1-fast-generate-001, which can synthesize visual motion alongside optional audio and dialogue. When a client application sends a generation request to the API, the service initiates a long-running operation that processes the media and outputs the final .mp4 video file directly to a destination bucket in Cloud Storage. Standard requests generate clips of 4, 6, or 8 seconds at 24 frames per second, with video extension capabilities that can lengthen videos up to a maximum duration of 30 seconds. System architectures must account for fixed quotas, which permit 10 requests per minute for Veo 2 preview models and up to 50 requests per minute for Veo 3.1 models.

Cross-Modal Embeddings with Multimodal Embedding Models

Multimodal embedding models convert text, image, and video data into joint high-dimensional vector representations within a shared semantic space. Services such as gemini-embedding-2 and multimodalembedding@001 map distinct input types into identical dimensions so that vectors from different modalities can be compared directly using distance metrics. When processing data with multimodalembedding@001, text inputs are limited to a maximum length of 32 tokens, while image inputs up to 20 megabytes are automatically resized to 512 by 512 pixels. Video assets stored in Cloud Storage can be processed in segments up to 2 minutes per request, using specific timestamp offsets to define the start and end boundaries. Engineers can configure prediction requests to reduce output dimensions or enable built-in features such as optical character recognition and audio track extraction.

Model Selection and Serving via Model Garden

Model Garden provides a centralized catalog within Vertex AI for discovering, customizing, and serving Google foundation models, partner models, and open-weight architectures. Organizations choose between fully managed Model-as-a-Service (MaaS) endpoints and self-hosted deployments for open models such as Llama 3.3, Llama 4 Maverick, Qwen3, GLM, and DeepSeek. Global endpoints offer higher availability and lower error rates for managed open-source models, while regional endpoints guarantee that data at rest remains strictly within designated geographic boundaries. To lower operational costs and improve throughput on self-hosted models, architectures can implement Prefix Caching to store repetitive prompt prefixes in host memory and Speculative Decoding to accelerate per-token generation speed. Retrieval-augmented generation (RAG) pipelines can integrate default publisher embeddings such as text-embedding-005 or open-source embeddings like e5-base-v2 directly from the catalog.

Custom Architecture Optimization with Neural Architecture Search

Vertex AI Neural Architecture Search (NAS) is an automated service that discovers custom neural network designs optimized for specific hardware, latency targets, and memory budgets. The service searches massive model design spaces—encompassing up to 10^20 potential structural variations—to identify high-performing architectures for specialized computer vision and machine learning tasks. NAS executes a structured two-stage optimization pipeline across distributed compute infrastructure:

• Stage-1 (Proxy Task Search): The service runs between 1,000 and 2,000 fast search trials across parallel GPU workers, training candidate structures on shortened epochs to score them against a chosen reward metric.
• Stage-2 (Full Training): The system selects the top candidate architectures identified during Stage-1 and trains them to complete convergence with full parameter scaling before production deployment.

Deploy, Secure, and Optimize Tuned and Model-as-a-Service Endpoints

Deploying Models to Scalable Vertex AI Endpoints

Vertex AI Endpoint resources host tuned adapter weights and foundation models on managed infrastructure to serve real-time predictions. Tuned models created via Parameter-Efficient Fine-Tuning (PEFT)—such as Gemma 3, Llama 3.1, or Qwen 2.5—are deployed to dedicated endpoints backed by provisioned compute instances, including NVIDIA A100-80GB or H100 GPUs. In contrast, standard foundation models like Gemini 2.5 Pro, Claude Opus 4, and Llama 3.1 405B can be consumed through serverless Model-as-a-Service (MaaS) APIs. Dedicated endpoints provide guaranteed throughput and predictable latencies for sustained enterprise workloads, whereas serverless MaaS endpoints scale automatically based on incoming traffic volume and bill strictly by token consumption.

Configuring Compute Resources and Dynamic Serving

Compute configuration requires matching machine types, memory tiers, and accelerator counts to specific model serving and concurrency requirements. Dedicated deployments can utilize high-memory machine types, such as A2 Ultra instances with fixed NVIDIA A100 GPUs, to support dynamic serving of multiple Low-Rank Adaptation (LoRA) adapters on top of a single shared base model. Within containerized Kubernetes environments, the GKE Inference Gateway manages traffic distribution across an InferencePool by inspecting real-time key-value cache utilization and active adapter states. The gateway routes requests to the most optimal model replica to maximize overall GPU utilization and minimize cold-start delays. This dynamic serving structure allows organizations to host multiple task-specific adapters on a single shared GPU footprint without paying for separate hardware pools.

Implementing Security, Safety, and Responsible AI Controls

Production inference architectures implement layered security controls across the network boundary, the data plane, and the model output layer. Network traffic between client applications and Vertex AI services or Vector Search indexes is isolated from the public internet using Private Service Connect (PSC) endpoints and VPC Service Controls perimeters. Data at rest across surrounding storage services, such as Cloud Run or Cloud SQL, is secured using customer-managed encryption keys (CMEK). At the application layer, engineers configure system instructions, content moderation filters, and input/output sanitization routines directly within the request pipeline. Responses are grounded against enterprise data sources and verified with automated citation checks to mitigate hallucinations and maintain regulatory compliance.

Optimizing for Performance and Cost at Scale

Scaling generative AI workloads requires balancing response latency, hardware efficiency, and computational expense across variable traffic patterns. Managed endpoints and containerized services on Cloud Run utilize GPU acceleration alongside prefix-cache-aware routing to eliminate redundant calculations across recurring prompt templates. To protect latency-sensitive workflows during peak traffic events, systems apply priority-based queueing policies where non-critical requests assigned a Priority value below zero are dropped. Infrastructure performance is continuously tracked through Cloud Logging and Cloud Monitoring, which record token counts, generation latencies, and error rates. Engineers evaluate these telemetry metrics to right-size provisioned GPU clusters, configure autoscaling thresholds, and optimize token usage across endpoints.

Execute Supervised Fine-Tuning and Parameter-Efficient Adaptation

Supported Gemini Models and Thinking Budgets for Fine-Tuning

Supervised Fine-Tuning (SFT) uses labeled input-and-output pairs to adapt foundational Gemini models to specialized formats, terminology, and domain-specific behaviors. The platform supports SFT across multiple foundation models, including Gemini 3.5 Flash, Gemini 3.1 Flash-Lite, Gemini 2.5 Pro, Gemini 2.5 Flash, and Gemini 2.5 Flash-Lite. When tuning models that possess reasoning capabilities, engineers should set the thinking budget to 0 for Gemini 2.5 models or select the MINIMAL thinking level for Gemini 3 models. Because the model internalizes the required response format directly from the training data, suppressing the thinking process during training lowers compute costs and improves inference speed. This tuning pattern is especially effective for structured tasks such as classification, entity extraction, query generation, and targeted summarization.

Dataset Preparation and Validation Constraints

Training datasets for supervised fine-tuning must be structured in JSON Lines (JSONL) format and uploaded to a Cloud Storage bucket prior to starting a job. Each JSONL line contains a single training example defining the prompt input and its corresponding expected target output. The platform enforces strict scale limits: text-only datasets can contain up to 10 million examples, multimodal datasets support up to 300,000 examples, and individual training files cannot exceed 1 gigabyte. Each individual example supports a combined maximum of 131,072 input and output tokens. A separate validation dataset may be supplied to monitor generalization, capped at either 5,000 examples or 30 percent of the total training set size when exceeding 1,000 validation examples.

Creating and Configuring Supervised Fine-Tuning Jobs

Engineers can launch supervised fine-tuning jobs using the Google Cloud console, the Google Gen AI SDK, the Agent Platform SDK for Python, the REST API, or Colab Enterprise. Job creation requires specifying a tuned model display name of up to 128 characters, selecting the base foundation model, defining the Cloud Storage dataset path, and setting the execution region. User data and tuned model artifacts are stored in the selected tuning region—limited to us-central1 and europe-west4—while the underlying computational training workload may be dynamically offloaded to available accelerators across the US or EU. The tuning process combines the newly trained adapter parameters with the original base model architecture while preserving the base model's broader capabilities.

Hyperparameter Selection for Parameter-Efficient Tuning

Supervised fine-tuning in Vertex AI utilizes parameter-efficient tuning, which freezes the core weights of the base foundation model and trains a lightweight adapter layer. Hyperparameter configuration centers on three primary parameters:

• Epochs: Controls the number of complete passes the training algorithm makes over the entire dataset, with the platform supplying an automated default based on total example count.
• Adapter Size: Determines the rank and total number of trainable parameters in the adapter, supporting values of 1, 2, 4, 8, and 16 for Gemini 2.5 and Gemini 3 models.
• Learning Rate Multiplier: Scales the optimization step size at each training iteration, which can be increased to accelerate convergence or decreased to mitigate overfitting.

Evaluation and Lifecycle Management of Tuned Models

When a fine-tuning job reaches a completed lifecycle state, the resulting tuned model artifact is registered in the Model Registry and deployed to a dedicated endpoint. The deployed model endpoint receives inference requests at a specific regional URI:

https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/endpoints/ENDPOINT_ID

Client applications must send production traffic directly to this unique endpoint URI rather than the standard base model API. During or after training, performance is evaluated by analyzing training loss curves, validation metrics, and automated quality metrics such as fluency using the Gen AI evaluation service. Continuous tuning features allow teams to iteratively train an existing tuned model or checkpoint with new data, saving intermediate checkpoints to deploy the highest-scoring candidate.

Quotas, Limitations, and Continuous Tuning

Fine-tuning environments operate under defined platform quotas, regional boundaries, and architectural limitations. Every project receives a default quota permitting at least one concurrent fine-tuning job globally across supported models, with additional concurrency requiring a quota increase request. Tuned model serving is restricted to US and EU multi-region endpoints, and customer-managed encryption keys (CMEK) are not supported for tuned model artifacts. Pricing for fine-tuning jobs depends entirely on the total volume of training tokens, which equals the number of dataset tokens multiplied by the configured epoch count. Subsequent inference calls to the tuned model endpoint incur standard per-token serving charges separate from the initial training cost.

Key Facts

• Supervised fine-tuning datasets are stored as JSONL files in Cloud Storage and support up to 10 million text examples or 300,000 multimodal examples, with a maximum token limit of 131,072 combined input and output tokens per example.
• Supervised fine-tuning job creation is supported in the us-central1 and europe-west4 regions, while tuned model serving is hosted on US and EU multi-region endpoints.
• Parameter-efficient fine-tuning for Gemini 2.5 and Gemini 3 models keeps base model parameters frozen and updates adapter layers configured with sizes of 1, 2, 4, 8, or 16.
• The multimodalembedding@001 model limits text inputs to 32 tokens, resizes images up to 20 megabytes to 512 by 512 pixels, and accepts Cloud Storage video segments up to 2 minutes in duration.
• Vertex AI Neural Architecture Search uses a two-stage pipeline consisting of Stage-1 proxy task searches across 1,000 to 2,000 candidate trials and Stage-2 full training of the top candidate models to convergence.

Frequently Asked Questions

What is the difference between dedicated Vertex AI endpoints and serverless Model-as-a-Service APIs?

Dedicated Vertex AI endpoints run on user-provisioned hardware instances with attached GPUs, providing predictable latency and support for custom LoRA adapters on sustained workloads. Serverless Model-as-a-Service APIs abstract underlying infrastructure management entirely, scaling compute dynamically and billing based on real-time token consumption.

Why should the thinking budget be set to zero or minimal when fine-tuning reasoning models?

Setting the thinking budget to 0 for Gemini 2.5 or selecting the minimal thinking level for Gemini 3 reduces training and inference costs while improving latency. The model learns the target task and output structure directly from the labeled examples during tuning, making prolonged internal reasoning steps unnecessary.

How does the GKE Inference Gateway optimize dynamic LoRA serving?

The GKE Inference Gateway tracks real-time key-value cache utilization and active LoRA adapters across model replicas in an InferencePool. It routes incoming inference requests to the specific worker containing the required adapter and cached prompt context, minimizing memory overhead and eliminating cold starts.

Where must client applications send inference requests after fine-tuning a Gemini model?

Applications must route inference requests to the dedicated tuned model endpoint URI generated upon deployment rather than the base foundation model API. The endpoint URI follows the standard Vertex AI format: https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/endpoints/ENDPOINT_ID.