Prepare and test your skills
Prepare and test your skills
Managed infrastructure services for the model lifecycle are cloud services that supply compute, storage, networking, and orchestration for AI workloads without manual hardware management. Vertex AI (opens in a new tab) is the managed platform that covers development, training, deployment, and inference, AI Hypercomputer is the integrated supercomputing system for AI and machine learning (ML) workloads, and Cloud Run GPUs (opens in a new tab) support AI inference. Together these services carry a generative AI application from prototyping through training, deployment, and monitoring.
Vertex AI managed resources are the notebooks, datasets, pipelines, containers, clusters, and endpoints that Vertex AI provisions and orchestrates without manual hardware setup. Development starts in Colab Enterprise, a collaborative managed Jupyter notebook environment with Vertex AI-managed compute and BigQuery integration, or in Vertex AI Workbench, a unified Jupyter notebook-based environment with managed scalable infrastructure and integrations to BigQuery and Cloud Storage. Teams store inputs in a managed dataset, a dataset object created in and hosted by Vertex AI, and run repeatable work in Vertex AI Pipelines, portable and scalable ML workflows based on containers.
Execution and deployment use containers, persistent compute, and endpoints:
AI Hypercomputer is the unified portfolio for Google Cloud AI infrastructure that vertically integrates hardware, software, and consumption models to maximize goodput, the measure of actual ML productivity. Performance-optimized infrastructure includes accelerators such as NVIDIA GPUs and Google Cloud TPUs plus networking and storage, open software includes optimized PyTorch, JAX, TensorFlow, and orchestration platforms, and consumption options include Flex-start VMs, Spot VMs, Reservations, and Dynamic Workload Scheduler. Dynamic Workload Scheduler provides atomic provisioning for entire blocks of interconnected nodes for large-scale training, while hyperparameter tuning optimizes hyperparameter settings within specified limits to maximize predictive accuracy.
GPUs are managed differently by lifecycle and maintenance model:
| GPU type | Lifecycle and maintenance | Reliability feature |
|---|---|---|
| General GPUs | Managed as independent compute units with asynchronous maintenance | Standard Google Cloud node auto-repair to maintain pod-level uptime for serving fleets |
| Clustered GPUs | High-performance clusters managed as a single tightly coupled system with synchronized maintenance | Automated hardware health monitoring and proactive node replacement |
Google Cloud TPUs use hardware designed for large matrix operations with on-chip high-bandwidth memory (HBM) for larger models and batch sizes. TPUs can be connected in groups called slices that scale workloads with little to no code changes. Teams deploy accelerators with Google Kubernetes Engine for automated container-native scaling, Slurm through Cluster Director, or Cluster Toolkit blueprints. In a cluster, a login node is the main entry point where a user accesses the cluster, submits jobs, and manages files, while worker nodes are individual machines that execute tasks.
Inference is running data points through an ML model to calculate an output, such as a single numerical score. Managed deployment puts a trained model on an endpoint that serves inferences, and Cloud Run GPUs run pre-trained or custom self-deployed models for AI inference from container images or source code.
| Inference type | How it works | Choose when |
|---|---|---|
| Batch inference | Takes a group of inference requests and outputs results in one file | Work can be processed as a group and results delivered together |
| Online inference | Obtains inferences on individual instances synchronously | Individual data points need real-time results |
Exam tip: Batch inference returns one file for a group of requests, while online inference returns synchronous results for individual instances.
Supporting services for reliable delivery provide the data, security, networking, file access, and observability that managed training and serving depend on. BigQuery is a fully managed, serverless, highly scalable enterprise data warehouse for analyzing massive datasets with SQL. Cloud Logging is a fully managed, real-time logging service that collects, stores, analyzes, and monitors logs from Google Cloud resources, on-premises applications, and custom sources. Network File System (NFS) is a client/server system that lets users access files across a network as if they were local.
Private connectivity and repeatable operations connect these pieces. Private Service Connect (PSC) maps private IPs in a customer network to another VPC network or to Google APIs, so traffic flows from the customer network to Google services without crossing the public internet. Infrastructure as Code (IaC) manages and provisions services through code, with configuration files containing infrastructure specifications for creating and editing infrastructure at scale. IAM permissions and service accounts control which principals and applications can access services and data.
Managed platforms reduce manual hardware management by shifting provisioning, orchestration, failure recovery, and optimization to the cloud provider. Managed Training on reserved clusters handles complex cluster configuration, hardware failure recovery, and framework optimization for the largest workloads. AI Hypercomputer maximizes goodput by optimizing scheduling and runtime layers so accelerators spend more time on productive computation, while optimized storage such as Google Cloud Managed Lustre and Cloud Storage Rapid removes I/O bottlenecks.
Choose managed infrastructure by workload shape and lifecycle need:
Specialized compute resources for AI workloads are cloud hardware and management services designed to train and serve large generative AI models efficiently. Cloud TPU (opens in a new tab) provides this capacity as TPU virtual machines that run ML workloads, including inference and large-model serving such as serving Qwen2-72B-Instruct with vLLM. Compute Engine is the service that creates and manages these TPU virtual machines as individual instances or as part of a managed group.
Cloud TPU virtual machines (VMs) are specialized compute instances for running ML workloads on Google Cloud. Compute Engine creates and manages Cloud TPU VMs either as single instances for isolated work or as members of a managed instance group (MIG) (opens in a new tab) for larger or replicated deployments. The MIG depends on Compute Engine for creation, scaling, and management, while the TPU VMs inside the group provide the specialized compute. Cloud TPU capacity can also be used in Google Kubernetes Engine (GKE), which gives teams another managed way to run TPU-backed workloads.
Single-host and multi-host TPU deployments differ in whether TPU VMs work independently or together on one workload. Single-host TPU deployments use individual Cloud TPU VMs that each operate independently inside a managed instance group, so each VM handles its own separate workload without depending on other VMs. Multi-host TPU slices are groups of interconnected Cloud TPU VMs that work together as a unit on a single workload, so data and computation flow across the connected VMs and the VMs depend on each other to finish the shared task.
| Deployment option | How the VMs work | Best for |
|---|---|---|
| Single-host TPU VMs in a MIG | Multiple independent TPU VMs, each handling its own workload | Multiple, independent workloads such as inference |
| Multi-host TPU slice in a MIG | Multiple interconnected TPU VMs working together as one unit | A single workload that needs combined capacity |
A single-host group scales by adding more independent TPU VMs, while a multi-host slice scales by using a larger interconnected unit for one workload. Matching the deployment shape to the workload keeps parallel inference efficient and gives large workloads the combined compute they require.
Exam tip: Use single-host TPU VMs in a MIG for multiple independent inference workloads, and use a multi-host TPU slice in a MIG when multiple interconnected TPU VMs must work together on one workload.
Infrastructure selection for generative AI is the choice of where and how models are trained, served, and connected to data. The main Google Cloud choices are Google Kubernetes Engine (GKE) (opens in a new tab), Vertex AI, and Cloud Run (opens in a new tab). These choices affect total cost of ownership (TCO), latency, and time-to-market because they change how much infrastructure the team manages, how quickly resources start, and how well workloads scale.
Cost, latency, and time-to-market effects come from different scaling, routing, and management models. GKE offers cost efficiency at scale through integration with Spot VMs and Flex-start VMs, and GKE Autopilot automates cluster operations and capacity right-sizing to reduce overhead. Cloud Run is a serverless platform that can scale to zero, so event-driven applications and smaller models can be served cost-effectively, while Vertex AI is fully managed and offloads infrastructure management to accelerate development and time-to-value for MLOps-focused teams.
Latency depends on routing, autoscaling, and acceleration. GKE Inference Gateway provides model-aware intelligent routing and load balancing for AI inference workloads, supporting low-latency online serving while controlling costs. Cloud Run automatically scales instances on demand and rapidly, and Cloud Run GPUs accelerate inference for computationally demanding models. Faster startup nodes on GKE reduce node startup times by up to 80% for GPU workloads, so training and serving start sooner.
Time-to-market depends on setup and operational load. Vertex AI is best for teams that want a fully managed, end-to-end platform to accelerate development. Cloud Run supports rapid deployment from prototypes in Vertex AI Studio, Google AI Studio, or Jupyter notebooks to scalable deployments with minimal configuration. GKE is best when deep control, portability, and a customized high-performance AI platform are needed, but it assumes more operational responsibility and Kubernetes expertise.
Platform options for generative AI trade management level and control for speed, scale, and operational work. GKE is Google-managed Kubernetes with deep control, Vertex AI is a fully managed end-to-end platform, and Cloud Run is a fully managed serverless platform for containers.
A decision tree branching on control, management level, and scaling needs to pick GKE for deep control and scale, Vertex AI for managed MLOps speed, or Cloud Run for serverless scale-to-zero inference.
| Platform | Management level | Control and flexibility | Best-fit workload | Cost, latency, and time-to-market implications |
|---|---|---|---|---|
| Google Kubernetes Engine (GKE) | Google-managed Kubernetes; Autopilot automates operations and right-sizing | Deep control, portability, custom pipelines, kernel-level customizations | Large-scale training and inference; standardized multi-tenant AI platform | Cost efficiency at scale with Spot and Flex-start VMs; low-latency serving with Inference Gateway; requires Kubernetes expertise |
| Vertex AI | Fully managed, end-to-end platform | Less infrastructure control; offloads management | MLOps and rapid time-to-value | Accelerates development and improves time-to-market |
| Cloud Run | Fully managed serverless platform | Containerized inference with scale-to-zero; less cluster control | Event-driven applications; serving smaller models cost-effectively; GPU inference | Scales to zero and scales rapidly on demand; rapid deployment from prototypes with minimal configuration |
Exam tip: Choose GKE for deep control, portability, and custom high-performance AI platforms; choose Vertex AI for fully managed MLOps and rapid time-to-value; choose Cloud Run for serverless containerized inference that scales to zero.
On-demand scaling lets capacity follow demand, while cost-optimized capacity controls spending on scarce accelerators. Cloud Run scales instances automatically and on demand and can scale to zero, which supports cost-effective serving for event-driven applications and smaller models. GKE supports cost efficiency at scale through Spot VMs and Flex-start VMs. For short-duration training, GKE flex-start provisioning mode powered by Dynamic Workload Scheduler improves the ability to secure scarce GPU and TPU accelerators. Kueue is a Kubernetes-native job queueing system that manages resource allocation, scheduling, quota management, and prioritization for batch workloads, which helps operational planning for large-scale AI projects.
Training and serving have different infrastructure needs across the AI lifecycle. For training and fine-tuning, GKE provides scale and orchestration to train very large models while minimizing costs. TPU multislice lets multiple TPU slices communicate over the Data Center Network (DCN) to achieve large-scale training. Ray on GKE abstracts Kubernetes infrastructure, letting teams scale large-scale data preprocessing, distributed training, and online serving with minimal code changes.
For inference and serving, GKE supports both GPUs and TPUs, and GKE Inference Quickstart provides benchmarked profiles for popular AI models to simplify performance analysis and deployment. Cloud Run can serve trained models as scalable HTTP endpoints, integrate with Vertex AI as a scalable frontend, and use Cloud Storage FUSE to mount a Cloud Storage bucket as a local directory so large model files are decoupled from the container. These combinations are not exclusive: Cloud Run can integrate with Vertex AI models, and GKE can run Ray, Kueue, and TPU multislice for distributed AI workloads.
AI Hypercomputer is the unified portfolio for Google Cloud AI infrastructure that vertically integrates hardware, software, and consumption models to maximize goodput, the measure of actual machine learning productivity. It maximizes goodput by optimizing scheduling and runtime layers so accelerators spend more time on productive computation, using performance-optimized infrastructure including NVIDIA GPUs and Google Cloud TPUs, open software including optimized PyTorch, JAX, TensorFlow and orchestration platforms, and consumption options including Flex-start VMs, Spot VMs, Reservations and Dynamic Workload Scheduler.
Batch inference takes a group of inference requests and outputs results in one file, while online inference obtains inferences on individual instances synchronously. Choose batch inference when work can be processed as a group and results delivered together, and choose online inference when individual data points need real-time results.
Single-host Cloud TPU virtual machines in a managed instance group each operate independently and handle their own separate workload without depending on other VMs, while a multi-host TPU slice in a managed instance group is a group of interconnected Cloud TPU VMs that work together as a unit on a single workload with data and computation flowing across the connected VMs. Use single-host TPU VMs in a managed instance group for multiple independent workloads such as inference, and use a multi-host TPU slice in a managed instance group when multiple interconnected TPU VMs must work together on one workload that needs combined capacity.
Choose Google Kubernetes Engine when deep control, portability and a customized high-performance AI platform are needed for large-scale training and inference, choose Vertex AI for a fully managed end-to-end platform for MLOps and rapid time-to-value, and choose Cloud Run for fully managed serverless containerized inference that scales to zero. Google Kubernetes Engine provides cost efficiency at scale through Spot VMs and Flex-start VMs and low-latency serving with Inference Gateway but requires Kubernetes expertise, Vertex AI offloads infrastructure management to accelerate development, and Cloud Run scales rapidly on demand and supports rapid deployment from prototypes with minimal configuration.
An enterprise development team is preparing to train and deploy generative AI models at scale. Currently, the engineers spend significant time manually provisioning compute clusters, configuring worker pools, and managing infrastructure scaling for model training and online inference endpoints.
What is a primary business and operational benefit of adopting a managed platform like Vertex AI for their machine learning lifecycle?
Last updated