Prepare and test your skills
Prepare and test your skills
Orchestrating integrated machine learning (ML) workloads means connecting accelerated infrastructure, orchestration software, and managed ML services into one automated pipeline. AI Hypercomputer (opens in a new tab) provides that foundation as an integrated supercomputing system that combines hardware, software, and consumption models. Vertex AI (opens in a new tab) manages the ML lifecycle stages of training, evaluation, and serving, while Cloud Run (opens in a new tab) provides serverless execution for event-driven steps and lightweight serving.
The integrated foundation for end-to-end pipelines is the vertically integrated AI Hypercomputer portfolio that covers prototyping through massive-scale training and real-time serving. The system combines three layers to maximize goodput, the measure of actual ML productivity. Performance-optimized infrastructure supplies accelerators, networking, and storage, open software supplies optimized frameworks and orchestration, and consumption options supply flexible provisioning. Because the three layers are designed together, scheduling and runtime overhead drops and accelerators spend more time on productive computation.
Accelerators and high-throughput storage in AI Hypercomputer are the NVIDIA GPUs and Google Cloud TPUs plus the storage systems that keep them fed with data. GPUs are divided by maintenance model: General GPUs are managed as independent compute units with asynchronous maintenance, while Clustered GPUs are managed as a single tightly coupled system with synchronized maintenance. Cloud TPUs are designed for large matrix operations and include on-chip high-bandwidth memory (HBM) for larger models and batch sizes, and TPUs connect in groups called slices that scale workloads with little to no code changes. High-throughput storage from Google Cloud Managed Lustre and Cloud Storage Rapid feeds data to those accelerators so input/output does not stall training or inference.
Reliability follows the same split. Clustered GPUs include automated hardware health monitoring and proactive node replacement for large training clusters, while General GPUs use standard node auto-repair to maintain pod-level uptime for serving fleets. Choose Clustered GPUs when distributed training needs synchronized nodes to stay together, and choose General GPUs when serving fleets need independent replacement without stopping the whole system.
Orchestration platforms and compute choices in AI Hypercomputer are the three ways to deploy and manage accelerators plus serverless execution. The open software layer provides optimized versions of PyTorch, JAX, and TensorFlow for these platforms:
The choice controls operating system management and scaling behavior. In GKE Autopilot mode Google manages nodes and capacity, while in Standard mode you manage node pools and GKE adds or removes containers with Horizontal and Vertical Pod Autoscalers and nodes with the cluster autoscaler. Cluster Director manages the cluster environment and lifecycle on custom Ubuntu, while Cluster Toolkit leaves the operating system self-managed but automates setup through versioned configuration files.
The following comparison shows how serverless, container, and cluster options differ for ML work:
| Compute option | Accelerators | Orchestration | Scaling |
|---|---|---|---|
| Cloud Run | NVIDIA RTX PRO 6000 Blackwell and L4, no TPUs | Serverless, Google-managed OS | Automatic based on requests or CPU consumption |
| Google Kubernetes Engine (GKE) | All available NVIDIA GPUs and all available Google TPUs | Kubernetes, batch jobs through Kueue | Automatic pod and node scaling |
| Cluster Director | NVIDIA GB300 Ultra, GB200, B200, H200, and H100, no TPUs | Slurm | Automatic based on CPU consumption |
Exam tip: Choose Cloud Run when the step needs serverless scale-to-zero without TPUs, and choose GKE when the workload needs all GPU and TPU types with container orchestration.
Vertex AI stages from training to inference are the managed steps that turn code into repeatable pipelines and deployed models. Vertex AI Pipelines defines portable, container-based ML workflows, while custom training jobs manage distributed training and hyperparameter tuning trials. After training, artifacts are stored in Cloud Storage buckets or registered in Model Registry for versioning, and parameters, code versions, dataset versions, and results are tracked with experiments. Automated evaluation checks thresholds such as accuracy, error rate, and F1 score, plus generative output quality, factual accuracy, and safety, before a model is promoted.
A left-to-right process flow showing Vertex AI Pipelines to custom training to storage and registry to evaluation gate to online and batch endpoints.
Deployment targets depend on traffic, latency, and cost tradeoffs. Vertex AI endpoints serve custom models and foundation models for online inference, which returns real-time results for individual requests, and for batch inference, which scores a group of requests into one output file. Ray on Vertex AI provides persistent clusters that remain allocated until deleted, so iterative development avoids startup overhead and distributed applications can request resources without managing virtual machines and containers directly. The serving layer depends on the training layer: only models that pass evaluation and registration flow to endpoints for production traffic.
Serverless execution and traffic distribution in Cloud Run and GKE decide where each request runs and how supporting data attaches. A GPU-enabled Cloud Run service hosts fine-tuned models for steps that need NVIDIA L4 or RTX PRO 6000 acceleration without operating a cluster. Memory and retrieval dependencies attach directly to the service: Memorystore for Redis supplies short-term memory, Firestore supplies long-term conversational history, and Cloud SQL for PostgreSQL or AlloyDB for PostgreSQL with the pgvector extension supplies vector search for Retrieval-Augmented Generation. Tools connect through Model Context Protocol servers, API calls, browser automation, and sandboxed code execution for external actions.
Traffic distribution decides where each request executes across those serving targets. GKE Inference Gateway places models behind a load balancer with model-aware routing that prioritizes GPU and TPU instances for generative AI and large language model inference. GKE custom compute classes define compute profiles with fallback policies, such as preferring reserved accelerators and falling back to Spot VMs when reserved capacity is unavailable. For diverse platforms, a centralized load balancer routes low-GPU requests to Cloud Run and complex GPU-intensive requests to GKE, with Cloud Service Mesh for policy management and a global external Application Load Balancer for low-latency global distribution.
Automated provisioning and consumption models in an integrated workload define infrastructure as code and select how capacity is obtained. Terraform defines GKE node pools optimized for training or inference, Vertex AI endpoints with machine types and scaling policies, and Cloud Storage buckets for data and artifacts. Cluster Toolkit provides curated blueprints for ready-to-use high-performance computing, AI, and ML clusters, while Config Connector manages Google Cloud resources as Kubernetes objects through YAML manifests. Cloud Build and Infrastructure Manager integrate those templates into continuous integration and continuous delivery pipelines with triggers on commits, validation with validate and plan, versioned artifacts, and separate development, staging, and production configurations.
Consumption options control cost and obtainability for the provisioned blocks. Dynamic Workload Scheduler (DWS) provides atomic provisioning of entire blocks of interconnected nodes for large-scale training, with Flex-Start mode for complex batch training up to seven days and Calendar mode for reservations up to three months on H100 and H200 GPUs and Trillium TPUs. Flex-start VMs, Spot VMs, and Reservations cover flexible, preemptible, and guaranteed capacity, while committed use discounts suit stable 24x7 inference or predictable training. Spot VMs reduce cost for fault-tolerant evaluation and experimentation but can be preempted, so high availability designs deploy inference across at least two virtual machines in different zones.
Manage Consumption Models and Scalability is the practice of choosing how accelerator capacity is obtained and how workloads grow or shrink to balance performance with cost. AI Hypercomputer (opens in a new tab) on Google Kubernetes Engine (GKE) (opens in a new tab) can use reservation-bound capacity, Spot VMs, and other consumption options, while schedulers and autoscalers place work and add replicas and nodes only when demand requires it. Reservation-bound capacity provides densely placed virtual machines (VMs) for tightly coupled training, while Spot VMs provide lower-priced capacity that lasts only for the life of the VM.
Consumption models and capacity placement define how GPU capacity is reserved and where those VMs land on the physical network. Reservation-bound capacity can provide densely placed capacity where VMs are allocated close together on the topology. Spot VMs with GPUs work like normal GPUs at lower spot prices but persist only for the life of the VM, and Spot VMs and other non-reserved consumption options are less likely to be allocated close together. Because Pod-to-Pod bandwidth depends on VM placement, placement directly affects execution time and cost for communication-heavy AI workloads.
The tradeoff between placement and cost-efficiency drives the choice of consumption model:
| Consumption option | Lifetime | Placement and performance effect |
|---|---|---|
| Reservation-bound capacity | Reserved capacity held for the workload | Densely placed VMs that support Topology Aware Scheduling with minimal network hops |
| Spot VMs with GPUs | GPU persists only for the life of the VM at lower spot prices | VMs less likely allocated close together, so Topology Aware Scheduling might not work well |
A serverless batch pattern provides another flexible option for bursty work. A Managed Service for Apache Spark batch workload can submit a parallelized PySpark job with five workers, where each worker is accelerated with NVIDIA L4 GPUs. This approach consumes accelerators for the duration of the batch job rather than holding a persistent training cluster.
Topology Aware Scheduling (TAS) is a scheduling approach that strategically places workers on the network topology to minimize communication overhead. TAS is used with Kueue, and it is especially important for large language model (LLM) training where gradient aggregation requires workers to communicate in a specific rank order. By placing sequentially communicating workers with fewer network hops between them, TAS reduces network contention and optimizes bandwidth utilization, which leads to faster convergence and shorter training times.
Preparing a GKE cluster for TAS follows an ordered setup where each step depends on the previous one:
TAS depends on densely placed capacity to succeed, so choose reservation-bound capacity when the goal is low-latency Pod-to-Pod communication. Choose Spot VMs and other flexible consumption options when the priority is lower cost for fault-tolerant or smaller work, including smaller-model training and single-host inference on G-series machines, with the understanding that scattered placement will limit TAS benefits.
Autoscaling inference workloads for throughput and cost scales Pods and nodes in response to model-server metrics. The vLLM server exposes metrics to Google Cloud Managed Service for Prometheus, and a Horizontal Pod Autoscaler (HPA) reads those metrics to decide when to add replicas. When the HPA decides to scale, GKE first schedules another Pod to deploy, which then triggers the node pool autoscaler to add a second node before it deploys the second vLLM replica. This chain means Pod scaling depends on the HPA decision, and node scaling depends on the pending Pod.
The choice of metric controls the balance between cost and latency. The num_requests_waiting metric measures requests waiting in the model server queue and grows noticeably when the KV cache is full. The gpu_cache_usage_perc metric measures KV cache utilization and directly correlates to the number of requests processed in a given inference cycle. Use num_requests_waiting when latency targets are achievable at maximum throughput and the goal is throughput and cost optimization, and use gpu_cache_usage_perc when queue-based scaling is not fast enough for latency-sensitive workloads. The averageValue target for the HPA must be determined experimentally for each model and traffic pattern.
Exam tip: Use num_requests_waiting to optimize for throughput and cost, and use gpu_cache_usage_perc for latency-sensitive workloads where queue-based scaling is too slow.
Optimize Hardware Acceleration with GPUs and TPUs is the use of specialized processors to increase training throughput and reduce inference latency for large ML workloads. AI Hypercomputer (opens in a new tab) moves data from storage to the training process, and as accelerators get faster, the speed of that data movement becomes increasingly important. Frameworks such as TensorFlow, open sourced by Google in 2015, run on Google Cloud with enterprise-grade support through TensorFlow Enterprise (opens in a new tab), which makes accessing data on Google Cloud faster and easier.
GPU selection for training and inference in Compute Engine (opens in a new tab) starts with the machine family, which determines how GPUs attach and what the virtual machine (VM) is best suited for. For accelerator-optimized A-series and G-series machine types, the GPU model is automatically attached to the instance, while N1 general-purpose machine types allow selected GPUs to be attached, except for N1 shared-core f1-micro and g1-small machines. G-series machines can also be used for training smaller models and for single-host inference.
The machine groups differ in attachment, models, and best use:
| Machine group | How GPUs attach | Example GPU models | Best for |
|---|---|---|---|
| A-series accelerator-optimized | GPU automatically attached | A4X Max with NVIDIA GB300, A4X with NVIDIA GB200, A4 with NVIDIA B200, A3 Ultra with NVIDIA H200, A3 Mega and A3 High and A3 Edge with NVIDIA H100, A2 Ultra with NVIDIA A100 80GB, A2 Standard with NVIDIA A100 40GB | Large-scale model training |
| G-series accelerator-optimized | GPU automatically attached | G4 with NVIDIA RTX PRO 6000, G2 with NVIDIA L4 | Smaller-model training and single-host inference, including graphics workloads with NVIDIA RTX Virtual Workstations (vWS) |
| N1 general-purpose | Selected GPUs attached by user | NVIDIA T4, NVIDIA P4, NVIDIA V100, with vWS variants for T4 and P4 | Flexible general-purpose VMs with acceleration |
GPUs can be added to Spot VMs at lower spot prices to reduce cost. GPUs attached to Spot VMs work like normal GPUs but persist only for the life of the VM.
GPU driver and CUDA stack configuration on Compute Engine requires NVIDIA drivers so applications can access the device. A GPU instance needs at least 40 GB of free disk space for the boot disk, and an alternative to manual setup is creating instances from images that already include drivers. For hardware-accelerated 3D graphics such as remote desktop or gaming, the system uses NVIDIA RTX Virtual Workstations drivers, while other workloads use the standard NVIDIA driver.
The software stack has three versioned components that can be mixed to suit the workload:
An earlier version of TensorFlow that works best with an earlier CUDA toolkit can run with a later NVIDIA driver required by the GPU, but the driver and toolkit versions must remain compatible. NVIDIA provides three driver branches:
For production workloads, use either the Production Branch or the Long-Term Support Branch. Installation guidance for those drivers is covered in the GPU driver documentation, so the driver choice depends on whether stability or latest-hardware support matters more.
Cloud TPUs are Google-designed accelerators for high-throughput training and serving across ML frameworks including TensorFlow and vLLM. A TPU request flows from the workload scheduler to a TPU host, where the requested TPU chips attach to the worker container through the google.com/tpu resource. A common single-host pattern is TPU v5e in a 2x4 topology, where one worker requests google.com/tpu: "8" with supporting CPU, memory, and ephemeral storage to feed the accelerators.
TPU workers depend on fast model and cache access because large language models can take significant time to download, load, and warm up on each new replica. Storage services such as Hyperdisk ML and Cloud Storage FUSE mounts provide the high-throughput data path that keeps TPU compute supplied. Choose TPUs when the framework supports them and the goal is maximum throughput for dense matrix operations in training or inference.
The optimized data path and worker placement for GPU training separate management traffic from high-performance GPU traffic. The management network carries external access such as SSH and most general network communication, while data networks carry high-performance communication between GPUs on different VMs, such as Remote Direct Memory Access (RDMA) traffic. Higher maximum transmission unit (MTU) values increase packet size and reduce packet-header overhead, which increases payload data throughput, so larger MTU values such as 8244 are recommended for these virtual private cloud (VPC) networks.
Placement of workers on that network topology directly affects distributed training time and cost. Topology Aware Scheduling (TAS) with Kueue strategically places workers to minimize communication overhead during gradient aggregation, which requires workers to communicate in a specific rank order. By minimizing network hops between sequentially communicating workers, TAS reduces network contention and optimizes bandwidth utilization, which leads to faster convergence and shorter training times. TAS works best with densely placed capacity obtained through reservation-bound capacity, while Spot VMs and other consumption options are less likely to be allocated close together, so TAS might not work well in that case.
Tuned inference autoscaling for throughput versus latency balances maximum throughput against response latency using server metrics from the model server. For vLLM serving on Google Kubernetes Engine (GKE), Google Cloud Managed Service for Prometheus ingests metrics in the vllm:metric_name format, and the Horizontal Pod Autoscaler scales the LeaderWorkerSet that runs the model replicas. Two metrics drive the scaling decision:
num_requests_waiting tracks the number of requests waiting in the model server queue and starts to grow noticeably when the key-value (KV) cache is full.gpu_cache_usage_perc tracks KV cache utilization, which directly correlates to the number of requests processed in a given inference cycle.Exam tip: Use num_requests_waiting when optimizing for throughput and cost, and use gpu_cache_usage_perc for latency-sensitive workloads where queue-based scaling is not fast enough.
The averageValue target for the chosen metric is determined experimentally, and scaling a replica triggers the node pool autoscaler to add capacity before the new vLLM replica deploys.
Use Cloud Run for serverless scale-to-zero steps that need NVIDIA L4 or RTX PRO 6000 acceleration without TPUs and without operating a cluster. Use Google Kubernetes Engine (GKE) for workloads that need all available NVIDIA GPUs and all available Google TPUs with Kubernetes orchestration and automatic pod and node scaling.
Reservation-bound capacity provides densely placed virtual machines that support Topology Aware Scheduling with minimal network hops for tightly coupled training. Spot VMs with GPUs work like normal GPUs at lower spot prices but persist only for the life of the virtual machine, are less likely to be allocated close together, and limit Topology Aware Scheduling benefits.
Use num_requests_waiting to optimize for throughput and cost when latency targets are achievable at maximum throughput, and use gpu_cache_usage_perc for latency-sensitive workloads where queue-based scaling is too slow. num_requests_waiting measures requests waiting in the model server queue and grows noticeably when the key-value (KV) cache is full, while gpu_cache_usage_perc measures KV cache utilization and directly correlates to the number of requests processed in a given inference cycle.
An AI research team is deploying multi-node large language model (LLM) fine-tuning jobs that take between 3 and 5 days to complete. The workload has the following architectural requirements:
Which AI Hypercomputer consumption option should you recommend?
Last updated