Prepare and test your skills
Prepare and test your skills
Vertex AI Pipelines (opens in a new tab) is the serverless orchestrator that runs the end-to-end machine learning lifecycle from data preparation to training, evaluation, and deployment. Cloud Build is the continuous integration and delivery engine that builds, tests, and packages ML code into versioned containers and pipeline definitions. Together they enable scheduled retraining, event-driven retraining on new data or drift, and safe promotion of code from development to production.
Automated pipeline execution means Vertex AI Pipelines runs start on a schedule or in response to an event instead of a person starting them manually. A scheduled run handles recurring work such as retraining on a fixed interval, while a triggered run responds to new data, code changes, or monitoring signals. Each run uses automated parameterization and versioning so inputs, code, and outputs are recorded for reproducibility.
Event-driven orchestration connects data and monitoring systems directly to pipeline triggers. A data ingestion workflow can trigger a training pipeline after BigQuery (opens in a new tab) ingests new data, so the model learns from the latest dataset without delay. Cloud Logging can publish a message when log filters match new-data arrival or when Vertex AI Model Monitoring emits skew and drift alerts, and that message starts a retraining or fine-tuning pipeline. Pipeline progress and resource usage flow to built-in logging and tracing and to Cloud Monitoring alerts, so failures and drift feed back into continuous training.
Continuous integration and delivery with Cloud Build means ML code changes are built, tested, and packaged into deployable pipeline artifacts automatically. Cloud Build imports source from Artifact Registry, GitHub, or Bitbucket, then executes a build defined in a cloudbuild.yaml file as a series of build steps where each step runs in a Docker container. Builds can start manually or through automated build triggers that fire when source changes are pushed, with separate routines for development and main branches.
A typical ML source repository contains all inputs Cloud Build needs to reproduce the pipeline:
cloudbuild.yaml scripts, test triggers, and pipeline deployment scriptssettings.yaml with pipeline input parametersCloud Build captures build-time values through substitutions such as $COMMIT_SHA, $REPO_NAME, $BRANCH_NAME, $TAG_NAME, $REVISION_ID, $PROJECT_ID, and $BUILD_ID. These substitutions let the same build definition tag images with the exact commit and trace an image back to the source change that produced it.
The Cloud Build workflow from source commit to pipeline deployment converts tested code into versioned containers and a compiled pipeline definition. The workflow starts when a developer pushes code from a local machine or Vertex AI Workbench to the development branch, which fires the Cloud Build trigger for that branch. The trigger only promotes code that passes tests:
A left-to-right process flow showing Cloud Build steps from dev-branch push through test gate, building images tagged $COMMIT_SHA, publishing to Artifact Registry, compiling pipeline.json, and running the deployed Vertex AI pipeline to produce a model.
/workspace directory in the Cloud Build runtime environment.$COMMIT_SHA.component.yaml file to point to the newly built images.pipeline.json definition file.pipeline.json to Artifact Registry for versioning and reuse.Testing and validation prove that infrastructure, dependencies, packaging, model quality, and data are ready for deployment. Stress tests with tools such as Locust, Grafana k6, or Apache JMeter check behavior under varying load, while latency, error rate, and resource use flow into Cloud Monitoring. Dataflow validates features and transformations, BigQuery supports large-scale data analysis, and Vertex AI Model Garden supplies base models for tuning generative AI systems.
Exam tip: Cloud Build triggers respond to code commits and produce versioned images and pipeline.json, while scheduled Vertex AI Pipelines runs and data or drift events trigger retraining on new data.
Execution governance for Vertex AI Pipelines means running each pipeline step with the right identity, network isolation, and compute resources. Pipeline identity uses dedicated Identity and Access Management (IAM) service accounts with least privilege, so a training pipeline can read from only a specific Cloud Storage bucket and write model artifacts to Model Registry. IAM Conditions add attribute-based control, for example allowing a service account to trigger a pipeline only if the request comes from a trusted Cloud Build trigger.
Network isolation keeps training data, code, and prediction traffic inside trusted boundaries. Training environments use dedicated projects or Virtual Private Clouds (VPCs) with firewalls and security perimeters, plus Shielded VMs or Confidential Computing for execution protection. Custom training jobs communicate privately with other Google Cloud services through a Private Service Connect interface and can use a private IP address for lower latency than a public IP. For online prediction, VPC Service Controls perimeters protect private endpoints, while the target Vertex AI service enforces IAM access to the model.
Compute optimization means selecting the machine type, accelerator, and scaling policy that fit each step:
Container strategy isolates tasks with different dependencies into different containers, so feature engineering, training, and inference each run separately and library conflicts do not spread. Steps can use validated prebuilt containers or custom containers, and artifacts stored in Model Registry get fine-grained access control, encryption at rest, and Cloud KMS protection.
Modular pipeline architecture means building the ML lifecycle as small, reusable steps that run in dependency order. An ML pipeline is a portable workflow based on containers and Google Cloud services. A pipeline component is a self-contained set of code that performs one step such as preprocessing or training. A pipeline job is an execution instance of a pipeline definition, represented by the PipelineJob resource as tasks connected by input-output dependencies.
Reusable components are defined with the Kubeflow Pipelines (KFP) software development kit (SDK) or TensorFlow Extended (TFX). The KFP SDK v2.0 or later uses a domain-specific language to define tasks and compile the pipeline into YAML. TFX v0.30.0 or later provides its own framework with built-in integration for Apache Beam and large-scale TensorFlow processing.
| Option | Version requirement | Choose when |
|---|---|---|
| Kubeflow Pipelines (KFP) | SDK v2.0 or later | For general use cases and custom frameworks such as PyTorch, scikit-learn, or XGBoost |
| TensorFlow Extended (TFX) | v0.30.0 or later | When the workflow uses TensorFlow to process terabytes of structured or text data |
Assembly chains reusable components from ingestion to deployment, with the orchestrator building a directed acyclic graph (DAG) so a training step runs only after its transformed dataset arrives:
Vertex ML Metadata automatically tracks which dataset, parameters, code, and component version produced which model and metrics. Lineage is stored under a MetadataStore, the regionalized top-level container for a project, so teams can reproduce a model version, compare runs, and audit history from ingestion to deployment.
Vertex AI (opens in a new tab) data integration is the process of moving data from operational source systems into storage and feature systems optimized for training and serving. BigQuery is the primary source for structured and semi-structured tabular data, while Cloud Storage is the primary source for image, video, audio, and other unstructured data. Data flows from source systems to BigQuery or Cloud Storage, then into a Vertex AI managed dataset or feature system that links data to models.
Vertex AI Feature Store is a managed service for centralized feature management and secure feature serving. A feature is a measurable attribute of an entity used to train a model or make a prediction. BigQuery serves as the offline store, so teams do not copy feature data into a separate offline store, and the same BigQuery source can feed both training and online inference while retaining historical data for retraining.
The feature registry, online store, and feature view divide management by role. The feature registry records which BigQuery tables or views are available for online inference. A feature group corresponds to one BigQuery source table or view. A feature view is a logical collection of feature columns materialized from that source into an online store and refreshed periodically. A FeatureOnlineStore instance is the online serving cluster that holds one or more feature views.
Data flows in order from BigQuery to online inference:
FeatureOnlineStore instance, choosing optimized online serving for ultra-low latency or Bigtable online serving for high data volumes.Data security combines resource-level IAM policies on stores and entity types, service-agent permissions limited to the specific BigQuery source, VPC Service Controls perimeters to limit exfiltration, and customer-managed encryption keys for RAG integrations. The simplest pattern keeps raw data and Vertex AI resources in the same project, while cross-project designs use IAM to grant controlled access.
Feature serving in Vertex AI Feature Store exports or fetches feature values through online and offline paths that serve different consumers. Online serving retrieves the latest values for a small subset of entities for real-time inference, while offline or batch serving exports high volumes of current and historical data with timestamps for training. For instances with optimized online serving, the service can perform vector similarity search to return approximate nearest neighbors based on embedding distance.
| Serving type | Data retrieved | Used when |
|---|---|---|
| Online serving | Latest feature values for a subset of entities from the online store | The application needs low-latency features for live prediction |
| Offline serving | High volumes of current and historical feature data from BigQuery | The workflow needs full datasets for model training or retraining |
When Vertex AI Feature Store is used as a vector database for RAG Engine, embedding vectors stored in BigQuery flow into the online store and RAG Engine queries them with low latency. Feature metadata is tracked in Knowledge Catalog, and embeddings support vector similarity search.
Exam tip: Choose optimized online serving for ultra-low-latency needs and Bigtable online serving for high data volumes; both use BigQuery as the offline source of truth.
Choosing training storage means separating ML data from operational systems so training does not interfere with production. Source data from service logs, devices, or scraped documents is extracted, reformatted, and stored in BigQuery or Cloud Storage. BigQuery is the recommended storage for all tabular data including intermediate processed data, stored as materialized tables instead of views or subqueries for maximum speed.
The Google Cloud tool to read BigQuery depends on the training framework:
| Training framework | Google Cloud tool to read BigQuery |
|---|---|
| TensorFlow with Keras | tf.data.dataset reader for BigQuery |
| TensorFlow Extended (TFX) | BigQuery client |
| Dataflow with Apache Beam | BigQuery I/O Connector |
| Other frameworks such as PyTorch, XGBoost, or scikit-learn | BigQuery import path for ONNX models |
Cloud Storage holds unstructured data in large sharded container formats that combine many small files into fewer large files to improve throughput. Use sharded TFRecord files when training with TensorFlow and Avro files for other frameworks. Aim for files of at least 100 MB, with between 100 and 10,000 shards, and use buckets and directories to group shards.
Preprocessing transforms raw storage into clean, validated features that match the training framework and model objective. BigQuery handles SQL-based preprocessing for structured data and is the most cost-effective method when transformations can be expressed in SQL. Dataflow built on Apache Beam handles large-volume batch and streaming transformations, including conversions of unstructured data to TFRecord for faster ingestion. Managed Service for Apache Spark reuses existing Spark logic serverlessly, while one-off Python or pandas scripts fit only smaller datasets that fit in memory.
Data quality checks validate schemas and detect anomalies before training. BigQuery supports automated quality scans for tabular data, Dataflow supports transformations not expressible in SQL, and TensorFlow Data Validation validates schemas and detects anomalies. For generative AI, quality also means accuracy, relevance, diversity, and alignment with required outputs, and BigQuery DataFrames can generate synthetic data when real data is insufficient.
A Vertex AI managed dataset links preprocessed data to custom-trained models through Vertex ML Metadata. A managed dataset provides descriptive statistics and supports automatic or manual splitting into train, test, and validation sets, with about 80% typically in training. Managed datasets are optional when a team wants full control over splitting in training code or when lineage is not critical.
Continue with this study guide: Preparing for Vertex AI data integration
AI Hypercomputer (opens in a new tab) is an integrated supercomputing system that combines performance-optimized infrastructure, open software, and flexible consumption models for ML workloads. Vertex AI manages the ML lifecycle stages of training, evaluation, and serving, while Cloud Run (opens in a new tab) provides serverless execution for event-driven steps and lightweight serving. Together they form one automated pipeline from prototyping through massive-scale training to real-time serving.
The integrated foundation for end-to-end pipelines is a vertically integrated portfolio designed to maximize goodput, the measure of actual ML productivity. Performance-optimized infrastructure supplies accelerators, networking, and storage, open software supplies optimized frameworks and orchestration, and consumption options supply flexible provisioning. Because the three layers are designed together, scheduling and runtime overhead drops and accelerators spend more time on productive computation.
Accelerators and high-throughput storage keep GPUs and TPUs fed with data. General GPUs are managed as independent units with asynchronous maintenance, while Clustered GPUs are managed as a single tightly coupled system with synchronized maintenance and automated health monitoring with proactive node replacement. Cloud TPUs are designed for large matrix operations, include on-chip high-bandwidth memory (HBM) for larger models and batch sizes, and connect in groups called slices that scale with little to no code changes. Google Cloud Managed Lustre and Cloud Storage Rapid provide the high-throughput data path so input/output does not stall training.
Orchestration platforms in AI Hypercomputer are the three ways to deploy and manage accelerators with optimized PyTorch, JAX, and TensorFlow. Google Kubernetes Engine (GKE) provides container-native scaling with Kubernetes orchestration. Slurm through Cluster Director provides Google-managed cluster environments with Slurm scheduling. Cluster Toolkit blueprints provide customizable Infrastructure as Code deployments in Slurm or GKE.
The choice controls operating system management and scaling behavior. In GKE Autopilot mode Google manages nodes and capacity, while in Standard mode teams manage node pools and GKE scales containers with Horizontal and Vertical Pod Autoscalers and nodes with the cluster autoscaler. Cluster Director manages the cluster lifecycle on custom Ubuntu, while Cluster Toolkit leaves the operating system self-managed but automates setup through versioned files.
| Compute option | Accelerators | Orchestration | Scaling |
|---|---|---|---|
| Cloud Run | NVIDIA RTX PRO 6000 Blackwell and L4, no TPUs | Serverless, Google-managed OS | Automatic based on requests or CPU consumption |
| Google Kubernetes Engine (GKE) | All available NVIDIA GPUs and all available Google TPUs | Kubernetes, batch jobs through Kueue | Automatic pod and node scaling |
| Cluster Director | NVIDIA GB300 Ultra, GB200, B200, H200, and H100, no TPUs | Slurm | Automatic based on CPU consumption |
Exam tip: Choose Cloud Run when the step needs serverless scale-to-zero without TPUs, and choose GKE when the workload needs all GPU and TPU types with container orchestration.
Vertex AI stages from training to inference turn code into repeatable pipelines and deployed models. Vertex AI Pipelines defines portable container-based workflows, while custom training jobs manage distributed training and hyperparameter tuning. After training, artifacts are stored in Cloud Storage or registered in Model Registry for versioning, and experiments track parameters, code, dataset versions, and results. Automated evaluation checks accuracy, error rate, F1 score, plus generative quality, factual accuracy, and safety before promotion.
Deployment targets depend on traffic, latency, and cost. Vertex AI endpoints serve custom and foundation models for online inference, which returns real-time results for individual requests, and for batch inference, which scores a group of requests into one output file. Ray on Vertex AI provides persistent clusters that remain allocated until deleted, so iterative development avoids startup overhead. Only models that pass evaluation and registration flow to endpoints for production traffic.
Serverless execution and traffic distribution decide where each request runs. A GPU-enabled Cloud Run service can host fine-tuned models with NVIDIA L4 or RTX PRO 6000 without operating a cluster. Memory and retrieval attach directly: Memorystore for Redis supplies short-term memory, Firestore supplies long-term history, and Cloud SQL for PostgreSQL or AlloyDB for PostgreSQL with pgvector supplies vector search for Retrieval-Augmented Generation. GKE Inference Gateway places models behind a load balancer with model-aware routing that prioritizes GPU and TPU instances, and a centralized load balancer can route low-GPU requests to Cloud Run and complex GPU-intensive requests to GKE.
Consumption models and capacity placement define how GPU capacity is obtained and where virtual machines (VMs) land on the physical network. Reservation-bound capacity provides densely placed VMs allocated close together on the topology. Spot VMs with GPUs provide lower spot prices but persist only for the life of the VM and are less likely to be allocated close together. Because Pod-to-Pod bandwidth depends on placement, placement directly affects execution time for communication-heavy AI work.
| Consumption option | Lifetime | Placement and performance effect |
|---|---|---|
| Reservation-bound capacity | Reserved capacity held for the workload | Densely placed VMs that support Topology Aware Scheduling with minimal network hops |
| Spot VMs with GPUs | GPU persists only for the life of the VM at lower spot prices | VMs less likely allocated close together, so Topology Aware Scheduling might not work well |
Topology Aware Scheduling (TAS) with Kueue strategically places workers on the network topology to minimize communication overhead. TAS is especially important for large language model training where gradient aggregation requires workers to communicate in a specific rank order. Preparing a GKE cluster for TAS follows an ordered setup:
Automated provisioning defines this infrastructure as code. Terraform defines GKE node pools, Vertex AI endpoints, and Cloud Storage buckets, while Cluster Toolkit provides blueprints for high-performance clusters and Config Connector manages Google Cloud resources as Kubernetes objects. Dynamic Workload Scheduler provides atomic provisioning of interconnected blocks, with Flex-Start mode for batch training up to seven days and Calendar mode for reservations up to three months on H100, H200, and Trillium TPUs.
GPU selection in Compute Engine starts with the machine family, which determines how GPUs attach. For A-series and G-series accelerator-optimized machines the GPU is automatically attached, while N1 general-purpose machines allow selected GPUs to be attached, except N1 shared-core f1-micro and g1-small. A-series with NVIDIA GB300, GB200, B200, H100, and A100 suits large-scale training. G-series with NVIDIA RTX PRO 6000 and L4 suits smaller-model training and single-host inference. N1 with NVIDIA T4, P4, and V100 suits flexible general-purpose VMs.
GPU instances require NVIDIA drivers and at least 40 GB of free boot disk space. The stack includes the NVIDIA driver for device access, the CUDA toolkit for building applications, and the CUDA runtime for execution. NVIDIA provides a Long-Term Support Branch prioritizing stability with R580 supported through June 2028, a Production Branch with latest-hardware support with R595 supported through March 2027, and a New Feature Branch not recommended for production. For production use either the Production Branch or Long-Term Support Branch.
Cloud TPUs accelerate high-throughput training and serving across frameworks including TensorFlow and vLLM. A TPU request flows from the scheduler to a TPU host, where chips attach to the worker container through the google.com/tpu resource. A common single-host pattern is TPU v5e in a 2x4 topology where one worker requests eight TPUs with supporting CPU, memory, and ephemeral storage. Storage such as Hyperdisk ML and Cloud Storage FUSE keeps TPU compute supplied, since large models take significant time to download and warm up.
The optimized data path separates management traffic from high-performance GPU traffic. The management network carries SSH and general communication, while data networks carry high-performance traffic such as Remote Direct Memory Access between GPUs on different VMs. Larger maximum transmission unit values such as 8244 increase payload throughput on VPC networks. For vLLM serving on GKE, Managed Service for Prometheus ingests num_requests_waiting, which grows when the KV cache is full, and gpu_cache_usage_perc, which tracks cache utilization. Use num_requests_waiting to optimize throughput and cost, and use gpu_cache_usage_perc for latency-sensitive workloads where queue-based scaling is too slow.
$COMMIT_SHA and a compiled pipeline.json.Vertex AI Pipelines is the serverless orchestrator that runs the end-to-end machine learning lifecycle from data preparation to training, evaluation, and deployment, while Cloud Build is the continuous integration and delivery engine that builds, tests, and packages ML code into versioned containers and pipeline definitions. Cloud Build triggers respond to code commits and produce versioned images and pipeline.json, while scheduled Vertex AI Pipelines runs and data or drift events trigger retraining on new data.
Use Kubeflow Pipelines (KFP) SDK v2.0 or later for general use cases and custom frameworks such as PyTorch, scikit-learn, or XGBoost. Use TensorFlow Extended (TFX) v0.30.0 or later when the workflow uses TensorFlow to process terabytes of structured or text data.
Online serving retrieves the latest feature values for a small subset of entities from the online store for low-latency live prediction, while offline or batch serving exports high volumes of current and historical feature data from BigQuery for model training or retraining. BigQuery serves as the offline store so teams do not copy feature data into a separate offline store, and feature views materialized into a FeatureOnlineStore instance provide the online path.
Choose Cloud Run when the step needs serverless scale-to-zero without TPUs using NVIDIA L4 and RTX PRO 6000 GPUs. Choose Google Kubernetes Engine (GKE) when the workload needs all available NVIDIA GPUs and all available Google TPUs with Kubernetes container orchestration and automatic pod and node scaling.
Last updated