Choosing CPU, GPU, TPU and Edge Hardware

Evaluating performance and cost trade-offs across Google Cloud hardware means matching workload shape to accelerator capability and then matching duration to a consumption model. Compute Engine (opens in a new tab) accelerator-optimized families span A4X Max, A4X, A4, A3 Ultra, A3 Mega, A3 High, A3 Edge, A2, G4, G2, and N1 with T4, complemented by Cloud TPUs (opens in a new tab) and CPU platforms. Throughput and latency for inference plus billing rules and discount limits determine total cost of ownership for training versus serving on Google Kubernetes Engine (GKE) (opens in a new tab) and Compute Engine.

1.1Accelerator-Optimized Machine Series for Training and Serving

Accelerator-optimized machine series for training and serving map workload intensity to GPU capability, with larger A-series systems for foundation-model work and G-series systems for cost-efficient inference. Pre-training uses only high-end A-series systems, while fine-tuning adds G4 and serving inference adds both G4 and A3 Edge. Graphics work uses smaller GPU systems, while high performance computing (HPC) can use any series based on offload.

The progression from pre-training to inference reflects falling interconnect and memory pressure as work shifts from synchronized large-scale updates to parallel request handling. Choose A4 and A3 variants when throughput depends on multi-GPU scale and high-bandwidth interconnects, and choose G4, G2, or N1+T4 when per-request latency is acceptable on smaller GPUs and budget dominates.

1.2Billing, Discounts, and Consumption Options

Billing, discounts, and consumption options for accelerator-optimized machines determine total cost of ownership alongside raw accelerator price. Google Cloud bills accelerator-optimized machine types for attached GPUs, predefined vCPU, memory, and bundled Local SSD if applicable. At any given point, a resource is eligible for only one type of discount, so on-demand, Spot, Flex-start, and reservation discounts do not stack.

Consumption for accelerator-optimized machines divides into four models with different discount behavior:

• On-demand: Provisioning with compute flexible committed use discounts (CUDs) only for eligible N1, G2, and G4 spend; for N1 with attached GPUs, flexible CUDs apply only to vCPUs, memory, and Local SSD disks, while for G2 and G4 they apply to vCPUs, memory, GPUs, and Local SSD disks.
• Spot: Spot VMs with automatic Spot discounts between 61% and 90%, where capacity can be preempted at any time and suits fault-tolerant, batch, or short-lived work.
• Flex-start: Flex-start provisioning through Dynamic Workload Scheduler (DWS) with automatic DWS pricing discounts, where resources provision as soon as capacity becomes available rather than immediately.
• Reservations: Reservations with resource-based CUDs purchased as resource-based commitments, where commitments for all GPUs and most Local SSD disks require attached reservations for those resources.

1.3Clustered GPU Capacity Models and Throughput Design

Clustered GPU capacity models and throughput design address large-scale training and multi-host inference where nodes must act as one tightly coupled system. AI Hypercomputer distinguishes General GPUs, managed as independent compute units with asynchronous maintenance and standard node auto-repair for serving fleets, from Clustered GPUs, managed as a single system with synchronized maintenance, automated health monitoring, and proactive node replacement. Cloud TPUs add hardware for large matrix operations with on-chip high-bandwidth memory (HBM) for larger models and batch sizes, and TPUs connect in groups called slices that scale workloads with little to no code changes.

Capacity for clustered GPUs favors dense allocation and stability over immediate availability:

Before creating a future reservation request for a supported GPU machine type, contact the account team or sales team, otherwise Google Cloud is likely to decline it. For complex topology needs, Cluster Director deploys and manages accelerator groups with colocation and scheduling for multi-GPU training on A3 Ultra H200 and A4 B200 across GKE and Slurm clusters.

1.4Inference Latency and Throughput Metrics

Inference latency and throughput metrics separate user-perceived responsiveness from system-level efficiency on GKE. Latency measures time to complete work for one request, while throughput measures tokens or requests completed across the server per second. Requests per second alone can mislead for large language models because throughput varies widely with context length.

Latency for inference workloads breaks into five related indicators:

• Time to First Token (TTFT) (ms): Time to generate the first token for a request.
• Normalized Time Per Output Token (NTPOT) (ms): Request latency normalized by output length, measured as request_latency / total_output_tokens.
• Time Per Output Token (TPOT) (ms): Average decode cost per token, measured as (request_latency - time_to_first_token) / (total_output_tokens - 1).
• Inter-token latency (ITL) (ms): Time between two output token generations, aggregated as mean, median, and percentiles such as p90, unlike TPOT which averages across the entire request.
• Request latency (ms): End-to-end time to complete a request.

Throughput for inference workloads breaks into four server-side indicators:

• Requests per second: Total requests served per second across the server.
• Output tokens per second: Measured as total_output_tokens_generated_by_server / elapsed_time_in_seconds.
• Input tokens per second: Measured as total_input_tokens_generated_by_server / elapsed_time_in_seconds.
• Tokens per second: Measured as total_tokens_generated_by_server / elapsed_time_in_seconds, counting both input and output tokens to compare workloads with high prefill versus high decode times.

Weight-loading and storage throughput directly affect startup latency and scale-out consistency. Cloud Storage FUSE mounts model weights for deployments with limited node scaling, Google Cloud Managed Lustre provides high-throughput low-latency file access from multiple Pods simultaneously and can attach to 10,000 or more Pods, and Google Cloud Hyperdisk ML provides dedicated block storage that can attach to up to 2,500 Pods for massive-scale low-latency access to large weights.

1.5Lowering Total Cost of Ownership While Protecting Performance and Availability

Lowering total cost of ownership while protecting performance and availability requires aligning stability needs with CUDs, Spot VMs, DWS modes, and model-aware routing. For stable 24x7 inference or training with fixed capacity, use committed use discounts for VMs and accelerators. For fault-tolerant evaluation and experimentation, use Spot VMs to reduce cost despite preemption, and use DWS to improve obtainability of high-demand accelerators.

DWS modes split by duration and reservation style: Flex-Start mode supports complex batch training needing high-end GPUs for up to seven days, while Calendar mode reserves specific GPUs such as H100 and H200 and TPUs such as Trillium for longer workloads up to three months. For fluctuating inference, vLLM can dynamically use TPUs and GPUs to balance capacity and performance needs. GKE Inference Gateway distributes traffic behind a load balancer with model-aware routing that prioritizes GPU and TPU instances for generative AI, using health checks and metrics from vLLM or Triton integrated into Cloud Monitoring via Google Cloud Managed Service for Prometheus.

Fallback and tiering further control cost without breaking latency objectives. GKE custom compute classes define compute profiles with fallback policies, for example prioritizing reserved GPU or TPU instances and falling back to Spot VMs when reserved capacity is unavailable. A centralized load balancer can route low-GPU requests to Cloud Run and GPU-intensive requests to GKE, while a global external Application Load Balancer distributes global user traffic for low latency.

1.6CPU Platforms and System Integration for Efficiency

CPU platforms and system integration for efficiency cover preprocessing, graphics-light serving, and orchestration choices that surround accelerators. Bare metal instances on 4th generation and later Intel Xeon Scalable Processors expose raw server resources plus function-specific accelerators where Intel QuickAssist Technology (Intel QAT) accelerates compression, encryption, and decryption, Intel Dynamic Load Balancer (Intel DLB) speeds data queues, Intel In-Memory Analytics Accelerator (Intel IAA) improves query processing, and Intel Data Streaming Accelerator (Intel DSA) accelerates data copy and movement. CPU platforms that support Confidential Computing can create Confidential VM instances to protect data while in use.

AI Hypercomputer vertically integrates performance-optimized infrastructure, open software, and consumption options to maximize goodput, the measure of actual machine learning productivity. Open software includes optimized PyTorch, JAX, and TensorFlow plus orchestration through GKE for container-native autoscaling, Slurm through Cluster Director or Cluster Toolkit blueprints, or Compute Engine managed instance groups for manual control. Highly managed deployment through Cluster Director, Cluster Toolkit, or GKE automates compute, networking, and storage setup, while GKE uses Container-Optimized OS by default and Compute Engine and Slurm clusters use accelerator-optimized images such as Rocky Linux and Ubuntu accelerator images alongside Deep Learning Software Layer container images with CUDA, NCCL, and ML frameworks.

Scalable serving architectures integrating specialized hardware combine heterogeneous compute, model-aware routing, and flexible capacity controls to serve inference reliably under fluctuating demand. AI Hypercomputer (opens in a new tab) provides the integrated foundation of accelerators, networking, storage, open software, and consumption models, while Google Kubernetes Engine (GKE) provides container-native scaling, load balancing, and fallback policies. Graphics processing units (GPUs) (opens in a new tab) handle heavy parallel inference, central processing units (CPUs) handle preprocessing and orchestration, and Dynamic Workload Scheduler (DWS) (opens in a new tab) improves obtainability of high-demand accelerators for elastic serving.

2.1Heterogeneous Compute Foundation for Serving

AI Hypercomputer is the unified portfolio for AI infrastructure that vertically integrates performance-optimized infrastructure, open software, and consumption options to maximize goodput, the measure of productive machine learning work. The infrastructure layer provides NVIDIA GPUs, Google Cloud TPUs, high-performance networking, and high-throughput storage, while the software layer provides optimized PyTorch, JAX, and TensorFlow with orchestration through GKE, Slurm through Cluster Director, or Cluster Toolkit blueprints. This integration lets a serving pipeline keep accelerators focused on productive inference instead of system overhead, storage stalls, or manual cluster setup.

Accelerator types in the foundation differ by maintenance model and scale behavior, which directly affects serving design. General GPUs are managed as independent compute units with asynchronous maintenance and standard node auto-repair to maintain pod-level uptime for serving fleets. Clustered GPUs are managed as a single tightly coupled system with synchronized maintenance, automated health monitoring, and proactive node replacement for dense multi-host inference and large-scale training. TPUs are hardware designed for large matrix operations with on-chip high-bandwidth memory (HBM) for larger models and batch sizes, and TPUs connect in groups called slices that scale workloads with little to no code changes.

Reliability and I/O complete the foundation for sustained inference throughput. Clustered GPU systems absorb hardware failures through system-level monitoring and replacement, while general GPU serving fleets depend on Kubernetes-level repair to keep pods available. High-throughput storage from Google Cloud Managed Lustre and Cloud Storage Rapid removes I/O bottlenecks that would otherwise starve accelerators during model loading and data feeding. Together, these choices let architects place compute-intensive generative AI and large language model (LLM) inference on accelerators while keeping lighter work and orchestration on more general compute.

2.2Optimize Availability and Cost for Inference Capacity

Optimizing availability and cost for inference capacity means selecting a consumption model that matches workload stability, duration, and tolerance for preemption. Stable 24x7 inference and predictable training use committed use discounts (CUDs) for virtual machines (VMs) and accelerators to lower price in exchange for commitment. Fault-tolerant evaluation and experimentation use Spot VMs for the deepest discount with the tradeoff that capacity can be preempted at any time. Workloads needing assured accelerator obtainability use DWS modes that persist and provision blocks of capacity rather than relying only on immediate on-demand requests.

Fungibility and abstraction further protect inference from accelerator shortages and topology complexity. A vLLM engine on GKE can dynamically use TPUs and GPUs to address fluctuating capacity and performance needs, so the same serving stack shifts between accelerator types as availability changes. Cluster Director deploys and manages accelerator groups with colocation and scheduling for multi-GPU systems such as A3 Ultra H200 and A4 B200 across GKE and Slurm clusters. Ray on Gemini Enterprise Agent Platform abstracts distributed infrastructure so applications request resources for training and serving without directly managing VMs and containers.

2.3Distribute Traffic with Model-Aware Load Balancing

GKE Inference Gateway is the load-balancing control point for inference with varying resource needs because it routes behind a load balancer using model-aware routing. Traffic flows from clients to the gateway, the gateway prioritizes instances with GPU and TPU accelerators for compute-intensive generative AI and LLM inference, and detailed health checks assess model status before sending requests. Serving frameworks such as vLLM or Triton expose LLM metrics, and those metrics flow through Google Cloud Managed Service for Prometheus into Cloud Monitoring for routing and autoscaling decisions. Observed signals include model queries per second (QPS), token throughput, and first token latencies from the model observability dashboard.

Additional routing patterns handle constrained accelerators, mixed platforms, and global users:

• Constrained GPU or TPU inference: traffic flows from the autoscaler to GKE custom compute classes, where a defined compute profile prioritizes reserved GPU or TPU instances first and then falls back to cost-efficient Spot VMs when reserved resources are temporarily unavailable.
• Mixed orchestration platforms: traffic flows from a centralized load balancer to Cloud Run for requests with low GPU needs and to GKE for complex GPU-intensive tasks, which improves cost and management efficiency for generative AI.
• Inter-service policy and observability: traffic flows between services through Cloud Service Mesh for communication and policy management, while logs flow to Cloud Logging and metrics flow to Cloud Monitoring for consistent operations.
• Global low-latency distribution: traffic flows from global users to a global external Application Load Balancer, which distributes load across multiple instances for high availability and low latency.

These patterns work together as one scaling loop where load balancing distributes traffic, utilization signals drive autoscaling, and fallback profiles preserve placement on suitable machines. When accelerator capacity tightens, custom compute classes keep critical inference on qualified hardware instead of failing or queuing. When demand spreads geographically or across platforms, centralized and global balancers keep latency low while Cloud Run absorbs lighter load and GKE absorbs accelerator-heavy load.

2.4Choose Deployment and Scaling Controls

Deployment choice for scalable serving is the decision between highly managed automation and manual control over VMs, environments, and orchestration. Highly managed deployment automates setup and orchestration of compute, networking, and storage to reduce operational overhead, using Cluster Director for automated cluster configuration, Cluster Toolkit for predefined and customizable blueprints such as A4 machine types with Slurm, and GKE for managed container orchestration with autoscaling and high availability. Less managed deployment leaves the team to create and manage VMs, custom environments, and orchestration layers when full control outweighs automation. The managed path is preferred for elastic online serving because autoscaling, health management, and compute-class fallbacks operate without manual node management.

GKE node configuration completes the hardware integration by defining the hierarchy used during scaling decisions. ComputeClasses let operators define prioritized node configurations so workloads run on selected hardware such as NVIDIA H100, A100, and L4 GPUs. During scale-out, GKE evaluates the hierarchy and provisions the highest-priority available profile that satisfies the workload, which enforces heterogeneous placement without manual node selection. Combined with model-aware gateway routing and centralized balancing, this hierarchy ensures that heavy inference lands on accelerators, lighter requests avoid wasting accelerators, and cost controls remain tied to actual resource needs.

Analyze workload characteristics to determine hardware suitability is the process of matching a model's compute pattern, memory footprint, and latency target to the processor that executes it most efficiently. AI Hypercomputer (opens in a new tab) provides the unified portfolio for this choice, combining performance-optimized infrastructure with accelerators, networking, and storage to maximize goodput. Graphics processing units (GPUs) (opens in a new tab) offload variable parallel computation from CPUs, Cloud Tensor Processing Units (Cloud TPUs) accelerate large matrix math, and accelerator-optimized machines (opens in a new tab) bundle GPUs with predefined vCPUs, memory, and Local SSD.

3.1Computational Demands That Define the Hardware Choice

Computational demands describe what the model stresses most: parallel math, memory capacity and bandwidth, and response time under load. Pre-training stresses sustained distributed throughput across many accelerators, fine-tuning stresses memory for weights plus optimizer state with less scale, and online serving stresses low-latency inference and consistent scale-out as request volume changes. Serving quality is measured with latency indicators such as Time to First Token (TTFT), Time Per Output Token (TPOT), inter-token latency (ITL), and end-to-end request latency, and with throughput indicators such as requests per second, output tokens per second, input tokens per second, and total tokens per second. A workload dominated by large matrix multiplications with large models and batch sizes points toward TPUs, a workload that must offload variable parallel computation points toward GPUs, and a workload that must respond in real time near users points toward inference-optimized GPUs and edge configurations.

3.2Cloud TPUs for Large Matrix Operations and Large Batches

Cloud Tensor Processing Units (Cloud TPUs) are accelerators designed for performing large matrix operations and include on-chip high-bandwidth memory (HBM) for larger models and batch sizes. The HBM keeps large weight matrices and activations close to the compute, so large-model training and generative AI inference sustain high throughput without stalling on off-chip memory access. Cloud TPUs can be connected in groups called slices that scale workloads with little to no code changes, so capacity grows by adding slice capacity rather than rewriting distribution logic. Choose Cloud TPUs when the dominant operation is dense matrix math and the workload benefits from large batches, distributed training for generative AI, and LLM inference served through engines such as vLLM that can dynamically use TPUs and GPUs.

3.3GPUs for Pre-Training, Fine-Tuning, and Inference Serving

Graphics Processing Units (GPUs) are the general parallel accelerators for AI workloads that must offload computation from CPUs, and Google Cloud organizes them into General GPUs and Clustered GPUs. General GPUs are managed as independent compute units with asynchronous maintenance and use standard node auto-repair to maintain pod-level uptime for serving fleets. Clustered GPUs are high-performance clusters managed as a single tightly coupled system with synchronized maintenance, automated hardware health monitoring, and proactive node replacement for large-scale training. Every accelerator-optimized virtual machine (VM) bundles its attached GPUs with predefined vCPUs, memory, and Local SSD where applicable, and the number of GPUs selected affects the CPU and memory capacity of the node.

Workload type determines the recommended accelerator-optimized machine series. Pre-training uses A4X Max, A4X, A4, A3 Ultra, A3 Mega, A3 High, and A2. Fine-tuning uses the same high-end series plus G4. Serving inference uses the broadest set, including A4X Max, A4X, A4, A3 Ultra, A3 Mega, A3 High, A3 Edge, A2, and G4. Graphics-intensive workloads use G4, G2, and N1 with T4, while high performance computing (HPC) can use any accelerator-optimized series with the best fit depending on the amount of computation that must be offloaded to the GPU.

3.4GPU Model Selection by Model Size and Task

GPU model selection refines the machine-series choice by model size and task, balancing memory and compute against cost and availability. NVIDIA A100 and H100 GPUs cover model fine-tuning and large-model inference where memory capacity and parallel throughput are highest. NVIDIA RTX PRO 6000 GPUs cover large, medium, and small model inference where flexible single-node serving matters. NVIDIA L4 GPUs cover large, medium, and small model inference with efficient inference per watt, while NVIDIA T4 GPUs cover medium and small model inference and graphics use where cost efficiency matters most.

The table shows a clear split by lifecycle stage: fine-tuning concentrates on A100 and H100, while inference spreads across all four options by model size. Choose A100 or H100 when fine-tuning or serving the largest models, choose RTX PRO 6000 or L4 when serving a mix of large to small models on GKE or Compute Engine, and choose T4 or L4 when the workload is medium to small inference or graphics rendering that does not need top-end memory.

3.5Edge and Low-Latency Options for Online Serving

Edge and low-latency serving options place inference capacity close to demand and minimize weight-loading and network delay. The A3 Edge machine series is included specifically in the serving-inference set alongside the core A3 and A4 families, so online endpoints that need accelerator inference with edge placement can stay within the same serving strategy. For deployments with limited node scaling, mounting model weights with Cloud Storage FUSE reduces startup complexity, while Google Cloud Managed Lustre supports high-throughput and low-latency file access from many pods simultaneously for consistent scale-out. Google Cloud Hyperdisk ML provides dedicated block storage for massive-scale scenarios demanding consistent, low-latency access to large model weights. Choose edge and inference-optimized GPUs such as L4, T4, G4, and A3 Edge when the workload characteristic is real-time fraud detection, recommendation engines, or other online serving where TTFT, TPOT, and request latency dominate the hardware justification.

Key Facts

• Compute Engine accelerator-optimized families map pre-training to A4X Max, A4X, A4, A3 Ultra, A3 Mega, A3 High, and A2, fine-tuning to those plus G4, and serving inference to those plus G4 and A3 Edge.
• A resource is eligible for only one discount at a time, and Spot VMs offer 61% to 90% discounts with preemption while Flex-start and calendar reservations offer up to 53% discounts with dense allocation.
• Cloud TPUs provide on-chip high-bandwidth memory (HBM) for large matrix operations and connect in slices that scale with little to no code changes.
• GKE Inference Gateway routes traffic behind a load balancer with model-aware routing that prioritizes GPU and TPU instances, using vLLM or Triton metrics via Google Cloud Managed Service for Prometheus.
• NVIDIA A100 and H100 GPUs are recommended for fine-tuning and large-model inference, while L4, RTX PRO 6000, and T4 focus on inference with T4 limited to medium and small models.