Training ML Models on Google Cloud

Organizing training data on Google Cloud means separating operational sources from training-optimized copies in BigQuery (opens in a new tab) and Cloud Storage (opens in a new tab), then registering those copies as versioned Vertex AI Datasets. Data flows from source systems such as service logs, devices, or web documents into BigQuery tables or Cloud Storage containers, then into a managed dataset that records schema, statistics, and lineage to custom-trained models. This separation protects production performance while giving training jobs high-throughput, reproducible inputs that can be reused across AutoML, custom training, and pipeline jobs.

1.1Separate Operational Sources From Training-Optimized Storage

Training-data preparation is the process of extracting data from live systems and converting it into formats and locations tuned for machine learning. Training jobs must not read directly from operational databases or live logs because training scans would contend with production traffic. The converted copy is stored in BigQuery for structured data or in Cloud Storage container files for unstructured data, so throughput, partitioning, and sharding can be tuned for distributed reads and tracked as artifacts.

1.2Organize Tabular Training Data in BigQuery

BigQuery ingestion is the standard path for structured and semi-structured tabular data intended for machine learning. All raw and most intermediate processed data stays in BigQuery tables organized by project, and training uses materialized tables instead of views or subqueries for maximum read speed. For large datasets, partitioning improves both query performance and cost efficiency because a query over a date range scans only matching partitions.

Data flows out of BigQuery to training through the BigQuery Storage Read API, and wrapping the source as a managed tabular dataset adds artifact tracking that links the table to the trained model. The reader that consumes the API depends on the training framework:

BigQuery supports ingestion-time, column-based, and integer-range partitioning. A column-based time-partitioned table flows automatically to the correct partition based on the column value, so training queries that filter on that column naturally prune unrelated partitions.

1.3Organize Unstructured Training Data in High-Throughput Cloud Storage Layouts

Cloud Storage ingestion is the standard path for image, video, audio, and other unstructured data. Cloud Storage is a globally scalable object store where data is stored as objects in containers called buckets. Individual media files are combined into large container formats before training, which improves read and write throughput compared to thousands of small objects. TensorFlow workflows use sharded TFRecord files, while other frameworks use Avro files for the same purpose.

Throughput depends on shard size, bucket location, and access path. Data flows from Cloud Storage objects through parallel download streams to each GPU or TPU worker, and each worker reads a disjoint subset of shards. Buckets and directories group the shards so a training job can list and stream a coherent prefix rather than discovering scattered objects.

Rapid Cache is an SSD-backed zonal read cache that accelerates reads to existing dual-region or multi-region buckets at up to 2.5 TB/s without code changes. Hierarchical namespace (HNS) enables a true file system structure with faster lookups and 8 times higher queries per second, and HNS is required for zonal buckets with Rapid Bucket. Cloud Storage FUSE mounts buckets as local file systems so frameworks read through the mount without refactoring, with architectures that can exceed 1 TB/s aggregate throughput when tuned with parallel downloads and client-side caching.

1.4Register Managed Vertex AI Datasets With Splits and Pipeline Reuse

A managed Vertex AI Dataset is a registered training input that creates a clear link between preprocessed data and custom-trained models. The dataset depends on the underlying BigQuery table or Cloud Storage objects, but adds descriptive statistics, centralized management, and integration with Vertex ML Metadata and Experiments to log parameters, metrics, and dataset-to-model artifacts. Choose a managed dataset when lineage or shared reuse matters; skip it when training code needs full control over splitting.

Standardized datasets are reused without duplication through an ordered lifecycle:

1. Prepare and store: Data flows from source systems to BigQuery tables or Cloud Storage sharded containers.
2. Register: The table, query result, or Cloud Storage manifest is imported as a managed Vertex AI Dataset.
3. Split: The dataset is divided into training, validation, and test subsets by automatic or manual assignment.
4. Consume: AutoML, custom training, or pipeline components read the named dataset version and record lineage to the model.
5. Compare: Experiments and Vertex AI TensorBoard track loss, accuracy, and dataset identity across runs.

Multimodal datasets extend the same pattern to generative AI workloads that mix text, image, audio, and video. A multimodal dataset is created once from BigQuery, DataFrames, or JSONL files in Cloud Storage and is stored in BigQuery, then serves both supervised fine-tuning and batch prediction with assemble and assess methods to preview data and estimate cost.

Ingesting data into training pipelines means matching each source to a landing service and a processing service based on volume, velocity, and format. BigQuery (opens in a new tab) is the landing zone for structured and semi-structured tabular data, while Cloud Storage is the landing zone for unstructured data such as text, images, video, and audio. Pub/Sub, Dataflow, and Dataproc move and transform data between operational sources and those training-optimized stores, and the training pipeline then reads from Cloud Storage or BigQuery rather than from live systems.

2.1Buffer High-Velocity Events Before Transformation

Pub/Sub is the fully managed real-time messaging service that decouples event producers from training ingestion when velocity is high or bursty. Data flows from publishers such as application logs, sensors, or external APIs to a Pub/Sub topic, and then from the topic to subscribers such as a Dataflow pipeline or Cloud Storage sink. Because the topic buffers messages independently of the subscriber, spikes do not overwhelm transformation jobs and components can scale separately.

For event-driven architectures, streaming ingestion combines with Eventarc to trigger Dataflow processing when new events arrive. Traffic flows from the event source to Eventarc to a Dataflow job, and only after the trigger fires does Dataflow read, transform, and write the micro-batch for ML use. Use this combination when retraining must react to new data rather than run only on a fixed schedule.

2.2Transform Batch and Streaming Sources With the Right Engine

Transformation engines for diverse sources divide work by interface and processing model. Dataflow runs both batch and stream processing for Apache Beam pipelines with ML-component integration, Managed Service for Apache Spark runs Spark workloads including Spark SQL, and BigQuery performs SQL-based transformation directly on staged tables. Raw data is first extracted from sources such as Cloud SQL, Spanner, AlloyDB for PostgreSQL, Firestore, Pub/Sub, Apache Kafka, or external APIs into training-optimized storage separate from the operational source.

BigQuery extract, load, and transform (ELT) loads raw data into a staging table first and then runs SQL queries that write output into a final production table. Choose in-flight Dataflow or Spark processing when formats differ widely or streams must be windowed and joined, and choose BigQuery SQL transforms when tabular data is already centralized.

2.3Validate Schemas, Files, and Distributions Before Training

Automated data validation is the continuous enforcement of schema consistency, format correctness, and statistical quality before data reaches training. TensorFlow Data Validation (TFDV) runs inside ingestion pipelines to validate schemas, verify formats and distributions, and monitor shifts over time. Validation results flow to BigQuery for analysis and to Cloud Storage for archiving, while monitored distributions trigger retraining when drift occurs.

File and storage validation determines whether inputs are complete and readable by the training service account. Cloud Storage data validation uses sizes and checksums to detect corruption, while format checks validate CSV, Parquet, TFRecord, Avro, and image payloads. Store unstructured data as large sharded containers of at least 100 MB with 100 to 10,000 shards, and use sharded TFRecord for TensorFlow and Avro for other frameworks.

2.4Centralize Features and Orchestrate Continuous Training Pipelines

Agent Platform Feature Store is the central location for creating, maintaining, sharing, and serving ML features from structured data. Feature data lives in a BigQuery table or view, including historical values for offline training and point-in-time correctness. To use the store, first create an online store instance and then define feature views, after which BigQuery serves as the offline store and the online store serves low-latency requests.

Orchestration with Vertex AI Pipelines automates every stage from data preparation to training, evaluation, and deployment as component tasks that pass data through arguments. The training stage depends on dataset size: Vertex AI Workbench fits small datasets or subsets, while the Vertex AI managed training service handles larger and distributed training with scheduling, hyperparameter tuning, and TensorBoard integration. Versioning covers code and components in Artifact Registry, datasets in BigQuery or Cloud Storage, and models in Model Registry, while Vertex ML Metadata and Knowledge Catalog track lineage for debugging.

Model training with different SDKs means choosing between serverless Vertex AI (opens in a new tab) custom training, managed AutoML, and containerized Kubeflow workflows on GKE based on control, problem fit, and scale. Vertex AI custom training runs user-supplied code on managed virtual machines (VMs) defined by a CustomJob, AutoML automates architecture and tuning for supported formats, and Kubeflow on GKE runs multi-step containerized workflows on Kubernetes. Training data flows from Cloud Storage or BigQuery into the training service, and trained outputs flow back to Cloud Storage and Model Registry.

3.1Package and Submit Vertex AI Custom Training Jobs

Custom container packaging is the method that bundles training code into a Docker image that Vertex AI runs directly. The workflow depends on Artifact Registry for hosting the image, Cloud Storage for staging data and outputs, and Identity and Access Management (IAM) for the service account identity. The prerequisite workflow runs in a fixed order:

1. Create Docker repository and config: Create an Artifact Registry Docker repository for the image.
2. Build custom container image: Build the image with training code and push the image URI.
3. Set up service account and buckets: Configure permissions and create buckets for inputs, TensorBoard logs, and outputs.
4. Create and launch custom job: Submit a CustomJob that references the image URI and monitor in TensorBoard.

A CustomJob carries a job_spec with one or more worker_pool_specs, and each pool defines a machine_spec, a replica_count, and either a container_spec or a python_package_spec. The Vertex AI SDK submits this specification to a regional endpoint such as us-central1-aiplatform.googleapis.com with a parent of projects/{project}/locations/{location}. A pre-built environment uses python_package_spec with an executor image and Python tarball, while a custom container uses container_spec with image_uri, command, and args for full control over libraries and startup.

3.2Configure Distributed Training and Accelerators for Custom Jobs

Distributed training configuration lives in worker_pool_specs, where each pool binds compute, accelerators, and code together. The machine_spec selects the VM family and accelerators, the replica_count selects how many identical VMs run, and the container or Python package selects what each replica executes. Scaling from one replica to many increases parallelism but requires code written to coordinate across workers.

Accelerator selection pairs machine_type with accelerator_type and accelerator_count on each VM. Training code must already be written to use graphics processing units (GPUs) or the GPUs remain idle. Vertex AI supports NVIDIA GB200, B200, H100 variants, H200, A100 variants, P4, P100, T4, V100, L4, and RTX PRO 6000 for serverless training. Using small machines such as n1-highmem-2 with GPUs can cause logging to fail because of central processing unit (CPU) constraints; selecting a larger machine type restores log delivery.

3.3Organize Containerized Training on GKE With Capacity Planning

Kubeflow on GKE is the approach for running containerized training as orchestrated, multi-step workflows on Kubernetes instead of single jobs. GKE provides the cluster control plane and accelerator node pools, while pipeline steps run as containers that read data and write predictions, checkpoints, and models to Cloud Storage. Platform teams manage clusters, quotas, and buckets, while data scientists submit container steps that request specific CPU, memory, and accelerator resources.

Capacity choices trade stability against cost and preemption risk:

Terraform provisions AI infrastructure declaratively, Cluster Toolkit provides curated blueprints for high-performance clusters, and Config Connector manages Cloud resources as Kubernetes objects. Experiment tracking uses Vertex AI Experiments for parameters and dataset versions, Model Registry for versioned artifacts, and Cloud Logging, Cloud Monitoring, Cloud Trace, and Cloud Profiler for health and bottlenecks.

3.4Automate Tabular Training With AutoML and SDK Job Classes

Vertex AI AutoML is a managed training environment that automates architecture search, preprocessing, hyperparameter search, provisioning, and packaging for supported data types. Data flows from BigQuery tables or managed datasets to the managed service, and the model flows to endpoints or edge devices. AutoML is appropriate when data matches a supported format, and for text, video, or tabular models the application must tolerate inference latencies greater than 100ms.

Structured-data AutoML in the SDK includes peer job classes such as AutoMLTabularTrainingJob for classification or regression, AutoMLForecastingTrainingJob for most forecasting, plus SequenceToSequencePlus and TemporalFusionTransformer variants for specialized forecasting. AutoMLTabularTrainingJob is configured with optimization_prediction_type such as regression and optimization_objective such as minimize-rmse, and run() takes the dataset, target_column, and split fractions to control flows into training, validation, and test partitions. Choose BigQuery ML when all data is in BigQuery and SQL plus an available model type fits, choose AutoML when automation outweighs control, and choose custom training when the problem does not match those criteria or must stay consistent with other platforms.

Troubleshooting training failures is the practice of finding why training cannot ingest data, why model code fails, or why compute and quota are insufficient. Data flows from source systems through Dataflow or SQL processing into Cloud Storage and BigQuery (opens in a new tab), then into training. A failure at any boundary can stall training, produce empty batches, or silently bias the model, so diagnosis must combine job logs, throughput metrics, file checks, schema validation, and lineage.

4.1Troubleshoot Data Pipeline and Input Failures

Automated data validation at ingestion checks data types, formats, ranges, and distributions before bad records reach training. TFDV generates a schema from training data and validates each new batch against that schema, while distribution tracking exposes drift that can trigger retraining. Validation results flow into BigQuery for historical analysis, and artifacts are archived in Cloud Storage for comparison to the last known-good schema.

Many input stalls are throughput or access failures rather than bad bytes. Data flows from Cloud Storage buckets to training workers, so the training service account depends on the bucket for read permission and the network path for sustained throughput. When jobs report permission denied, missing objects, or slow reads, check least-privilege service accounts, bucket-level IAM policies, and Cloud Audit Logs streamed to Cloud Logging for denied calls or revoked access. Continuous quality assessment in Knowledge Catalog tracks accuracy, freshness, completeness, validity, and consistency, and lineage shows which pipeline run produced the bad partition.

4.2Diagnose Dialogflow to Insights Notification Failures

Analyze and remediate model-specific runtime failures by confirming whether a conversational runtime is emitting expected signals and isolating where the signal path breaks. For a Dialogflow integration with Customer Experience Insights, remediation follows an ordered diagnostic flow from metrics to logs to escalation. The flow depends on notifications moving from the Contact Center AI Runtime source to the Customer Experience Insights Environment destination when conversations transpire.

1. Verify notifications: Check Cloud Monitoring with Metrics Explorer for notification activity from Dialogflow to Customer Experience Insights.
2. Isolate the failure point: Check Logs Explorer to find where the integration went wrong.
3. Escalate with evidence: Contact a Google representative with diagnostic API outputs if prior steps do not resolve the issue.

Verifying runtime notifications uses Resource Type Customer Experience Insights Environment and Metric Notifications From Contact Center AI Runtime. If the metric shows no notifications during active conversations, the failure is in the notification path, not downstream analysis. Escalation requires three configuration queries for securitySettings, agents/AGENT-ID, and conversationProfiles, each sent with Content-Type: application/json, Authorization: Bearer token, and x-goog-user-project to LOCATION-dialogflow.googleapis.com.

4.3Diagnose Resource Allocation and System-Level Failures

Diagnose resource failures by using logs and metrics to prove whether a job failed because compute, memory, disk, or quota was insufficient. In Managed Service for Apache Spark, training data flows from storage through executors on worker nodes under YARN, while shuffle data flows to attached disks, so pressure at any stage can kill executors or the driver. Cloud Logging carries failure signatures and Cloud Monitoring carries resource signals, and the two must be interpreted together.

Out-of-Memory failures appear as Lost executor, java.lang.OutOfMemoryError: GC overhead limit exceeded, and Container killed by YARN for exceeding memory limits. Shuffle fetch failures often follow because crashed executors leave no server to answer fetches, producing FetchFailedException or Failed to fetch. Fixes change where memory is granted: raise spark.executor.memoryOverhead and lower spark.executor.memory for PySpark, move to high-memory machine types, or use smaller partitions to reduce bytes per task.

Disk pressure causes slow shuffles and UNHEALTHY YARN nodes when shuffle directories exceed 90% utilization. Standard persistent disks should be at least 1 terabyte per worker because throughput scales with size.

Memory Protection terminations occur when the system kills a process to avoid a node hang, signaled by Driver received SIGTERM/SIGKILL signal and exited and confirmed by the google.dataproc.oom-killer log or dataproc.googleapis.com/node/problem_count metric. For driver kills, increase driver-required-memory-mb or lower concurrent jobs; for worker kills, verify YARN container sizes, consider lowering yarn.nodemanager.resource.memory-mb, or repartition to fix skew.

Hyperparameter tuning on Google Cloud means defining which hyperparameters Agent Platform may vary, what values each may take, and which search algorithm chooses the next trial under a fixed budget. A HyperparameterTuningJob runs many trials of the same training application with different values, reports a numeric metric from each trial, and returns the most effective configuration plus a per-trial summary. Hyperparameters such as learning rate or layer count stay constant during a single trial, while model parameters such as neural network weights change during training.

5.1Parameterize Search Spaces With Discrete, Continuous, and Categorical Types

The search space in a ParameterSpec is parameterized by choosing one value specification that controls data type, allowed range, and sampling behavior. Each additional hyperparameter expands the space and can increase the number of trials needed, and because charges are based on trial duration, a tight space directly controls time and cost. Trial values flow from the StudySpec to training code as command-line arguments, so the module must define a matching argument for each hyperparameter, and metrics flow back through the cloudml-hypertune package.

In BigQuery ML, tuning is enabled by NUM_TRIALS, with HPARAM_RANGE for continuous bounds and HPARAM_CANDIDATES for discrete values. A ConditionalParameterSpec adds hyperparameters only when a parent value matches, such as adding num_hidden_layers only when training_method is DNN. A shared child lets lessons transfer across branches, while separate children with the same name keep tuning independent.

5.2Scale Continuous Spaces and Choose a Search Algorithm

Scaling in a ParameterSpec reshapes how Agent Platform samples DOUBLE and INTEGER types by mapping the interval to 0 through 1. UNIT_LINEAR_SCALE preserves equal numeric distances, UNIT_LOG_SCALE spreads out small values and requires a strictly positive space, and UNIT_REVERSE_LOG_SCALE spreads out values near the top and also requires a strictly positive space. Scaling matters most when a hyperparameter spans orders of magnitude, such as a learning rate from 0.0001 to 1.0.

The search algorithm controls how the next configuration is selected from prior results. If no algorithm is specified, the default Bayesian optimization models the relationship between hyperparameters and the goal and balances exploration against exploitation.

Trial budget translates the choice into cost control. maxTrialCount caps total trials, parallelTrialCount caps concurrent trials, and maximum failed trial count stops the study early after too many failures. More trials generally improve results but increase time and cost, while parallel trials shorten wall-clock time but can reduce sequential Bayesian benefit.

5.3Analyze Results and Automate Tuning in Pipelines

Trial results record how each configuration performed against a studySpec goal of MAXIMIZE such as accuracy or MINIMIZE such as loss. When the job finishes, the platform reports the best configuration and a per-trial summary, which is the basis for judging whether gains came from a meaningful region or from noise or overfitting. Vertex AI Experiments logs parameters, code, dataset versions, and results, while TensorBoard visualizes curves and hyperparameter relationships.

Vertex AI Pipelines embeds tuning into continuous workflows as modular components that pass artifacts in order:

1. Data preparation and validation occur first and produce the versioned dataset.
2. Training and tuning run next and produce the best model and trial summary.
3. Evaluation checks thresholds after tuning and decides whether the candidate can advance.
4. Deployment and monitoring follow only after evaluation passes.

The tuning component HyperparameterTuningJobRunOp creates a HyperparameterTuningJob with a studySpec and a trialJobSpec for machine type, accelerators, and container or Python package. A custom container for tuning must install cloudml-hypertune, parse each hyperparameter with argparse, and pre-install CUDA and cuDNN for GPU training. Cloud Composer orchestrates broader cross-service DAGs, while Vertex AI Pipelines orchestrates ML-native workflows; hyperparameter tuning jobs in this loop do not support TPUs.

5.4Constrain Serving Cost With Autoscaling and Gateway Choices

Vertex AI prediction autoscaling allocates replicas between MIN_REPLICA_COUNT and MAX_REPLICA_COUNT to hold a per-replica metric near its target. Traffic flows from the endpoint to deployed replicas, and the autoscaler adds or removes replicas based on CPU utilization, GPU duty cycle, DCGM GPU utilization, vLLM cache usage, waiting requests, request count, or Pub/Sub queue size. CPU and GPU utilization default to 60 percent while other metrics are disabled by default, and a deployment configured to scale only on CPU will not scale when GPU usage is high.

GKE Inference Gateway (opens in a new tab) routes generative AI traffic using KV cache utilization, queue length, GPU utilization, and LoRA adapter availability with request criticality and prioritization. GKE Gateway with Custom Metrics balances general HTTP traffic on ORCA-reported signals with WEIGHTED_ROUND_ROBIN policies. Choose Inference Gateway when routing needs LLM-specific signals or LoRA affinity, and choose Custom Metrics when one or two uniform-cost signals are sufficient.

Fine-tuning foundation models adapts a proprietary Gemini model or an open-weight model such as Llama 3.1 to a downstream task. Data flows from JSON Lines (JSONL) datasets in Cloud Storage into a managed tuning job, and the job writes a tuned model, checkpoints, and experiment metadata to a Cloud Storage output directory and deploys a servable endpoint. The tuned adapter does not replace the base model; it attaches task-specific behavior to frozen base weights while endpoint infrastructure remains managed.

6.1Choose a Customization Strategy by Task, Data, Latency, and Cost

Model customization is a spectrum from prompt engineering to grounding to tuning that trades complexity, data, latency, and cost against depth of adaptation. Prompt engineering crafts instructions for a frozen Gemini model and is best for rapid prototyping with limited labeled data. Grounding with Retrieval-Augmented Generation (RAG) retrieves chunks from vector stores or APIs into the prompt at inference time, keeping proprietary knowledge outside weights for freshness without retraining.

Supervised fine-tuning (opens in a new tab) teaches a new skill by training to predict the correct output for each labeled input, adjusting weights to minimize prediction error. Supervised tuning is a good option when output is well-defined with labels, such as classification, entity extraction, extractive question answering, strict summarization formats, or persona control. For code models, supervised tuning is the only supported tuning option.

Parameter-efficient tuning updates only a small subset through Low-Rank Adaptation (LoRA), where larger adapter size learns more complex tasks but needs more data and time. Distillation uses a larger teacher to label prompts for a smaller student and is most effective for transferring reasoning such as math or chain-of-thought. Preference tuning builds on supervised tuning with human feedback for subjective alignment.

6.2Prepare Supervised Datasets and Configure Training Hyperparameters

Supervised dataset preparation stores training and optional validation examples as JSONL files in Cloud Storage and points the tuning job at their gs:// URIs. Each line contains one example, and best results use at least 100 to 500 high-quality examples that reflect production distribution and format. For distillation, training data can be prompts only because the teacher supplies completions, but validation data must include ground-truth completions.

Text datasets use prompt-completion objects with prompt and completion, turn-based chat objects with user and assistant messages, or GenerateContent objects with systemInstruction and contents including multimodal file_data. Multimodal images can be referenced by Cloud Storage URI or base64 in JPEG, PNG, and WEBP formats. Training token cost for Gemini supervised tuning is tokens multiplied by epoch count, with up to 131,072 input and output tokens per example.

Training hyperparameters control passes over data, trainable parameters, and step size. Epochs define complete passes with a recommended default from dataset size, adapter size controls trainable adapter parameters with supported sizes including 1, 2, 4, 8, and 16, and learning rate multiplier scales the recommended step size with a default of 1. Submit the first job with defaults, then adjust only to fix recurrent errors after evaluating failures.

6.3Execute Tuning Jobs and Manage Checkpoints and Artifacts

A fine-tuning job binds a base model, datasets, hyperparameters, and output location into a managed pipeline that produces a tuned model, endpoint, and experiment record. Supported Gemini bases include Gemini 3.5 Flash, Gemini 3.1 Flash-Lite, Gemini 2.5 Pro, Gemini 2.5 Flash, and Gemini 2.5 Flash-Lite, while open-model tuning can start from a base such as meta/llama3_1@llama-3.1-8b or a same-architecture custom checkpoint for continuous tuning. A console job defines the model in Agent Platform Studio, sets epochs and adapter size, attaches training and validation JSONL paths, starts tuning, then records the tuned model name, endpoint, and experiment context.

The same job can run programmatically with the Gen AI SDK client.tunings.tune, the Vertex AI SDK sft.train with FULL or PEFT_ADAPTER mode, or REST tuningJobs.create with training_dataset_uri, validation_dataset_uri, and epochCount, adapterSize, and learningRateMultiplier. Tuned artifacts preserve progress so the best version can be selected: a maximum of 10 checkpoints are stored, with one per epoch when epochs are fewer than 10. Open-model outputs include distillation_labelled_dataset.jsonl and postprocess/node-0/checkpoints/ with final/ and intermediate checkpoint directories.

6.4Evaluate, Deploy, and Integrate Tuned Adapters With Agents

Automated evaluation of a tuned model uses managed primitives to score outputs against quality and safety criteria without manual review of every response. The Gen AI evaluation service evaluates one model or application against judgment criteria including RAG and agent optimization, while the automatic side-by-side (AutoSxS) tool performs pairwise comparison of two models on the same prompts to choose a winner. Evaluation data flows from BigQuery or Cloud Storage into the service, and Agent Platform Pipelines automates filtering, augmentation, and human-in-the-loop review.

Deployment registers the tuned artifact as a versioned model and releases it to a managed endpoint with monitoring and rollback:

1. Register version: Store the adapter and metadata as a new version in Model Registry with lineage.
2. Validate in CI/CD: Run Cloud Build tests plus stress tests and bulk evaluations.
3. Release in controlled stages: Deploy to Inference with staged validation and quick rollback.
4. Enable continuous training: Schedule pipeline runs on new data with current base models.

After deployment, Model Monitoring watches for data drift and shifts in output quality and safety compliance. Integration connects the endpoint to Vertex AI Agent Builder as the reasoning engine: the agent sends prompt and context to the tuned endpoint, receives a domain-adapted completion, and routes it through tool calls and RAG grounding before responding.

Key Facts

• Training copies live separate from operational sources, with structured data as materialized BigQuery tables read through the BigQuery Storage API and unstructured data as sharded TFRecord or Avro containers of at least 100 MB with 100 to 10,000 shards in Cloud Storage.
• Vertex AI custom training runs user code as a CustomJob where each worker pool entry binds machine_spec, replica_count, and container_spec or python_package_spec, and GPU pools must use A2, N1, or G2 machine types.
• Dynamic Workload Scheduler Flex-Start supports batch training up to 7 days without a fixed start date, while Calendar mode reserves H100, H200, and Trillium capacity up to 3 months.
• GRID_SEARCH requires all parameters to be INTEGER, CATEGORICAL, or DISCRETE and cannot be used with a DOUBLE range, while maxTrialCount, parallelTrialCount, and maximum failed trial count bound tuning cost.
• Supervised tuning requires 100 to 500 high-quality labeled JSONL examples in Cloud Storage, with distillation training data allowed to be prompts only but validation requiring ground-truth completions, and a maximum of 10 tuning checkpoints are stored.