Cloud TPU topologies define how TPU chips are physically arranged and connected within Google Cloud. The topologies use a three-dimensional format like 4x4x4, where each number represents the chip arrangement along a dimension. For large TPU deployments, the dimensions must follow an A ≤ B ≤ C ordering pattern, and for configurations exceeding 64 chips, each dimension must be a multiple of four. The maximum topology reaches 12x16x16 for TPU v4 and 16x16x24 for TPU v5p. These chips communicate through Inter-Chip Interconnect (ICI) links, which provide the high-speed connections needed for distributed training.
TPU node pools in Google Kubernetes Engine (GKE) can be configured as either single-host or multi-host slices. A multi-host slice connects multiple virtual machines together, with their local TPU chips communicating through high-speed ICI links to enable unified tensor execution across the entire slice. When any virtual machine in a multi-host slice needs repair, GKE shuts down all virtual machines in that slice and evicts the running Pods until replacement hardware is ready. A single-host slice consists of independent virtual machines where each TPU communicates only through the standard Data Center Network, without direct ICI links between hosts.
Accelerator consumption models determine how compute resources are acquired and managed on Google Cloud. Organizations can provision on-demand resources at full price, use Spot VMs for discounted pricing (with a 30-second warning before preemption), or use Flex-start provisioning for workloads running up to seven days. For predictable workloads, calendar-mode future reservations secure capacity for up to 90 days, while long-term TPU reservations commit to one year or longer. Hardware faults such as High Bandwidth Memory error-correcting code failures or ICI network degradation trigger core dumps and require node replacement.
JetStream is an open-source serving engine that optimizes inference for large models on TPUs and GPUs within GKE. JetStream integrates with PyTorch/XLA and JAX to deploy large language models efficiently. The framework uses continuous batching to group incoming requests dynamically, reducing latency and increasing throughput. Memory usage is reduced through Int8 weight quantization (converting 32-bit weights to 8-bit integers) and Key-Value cache quantization (compressing attention mechanism states).
Google Distributed Cloud (GDC) connected extends deployment options to on-premises environments with dedicated hardware in rack or server form factors. These configurations support NVIDIA L4 GPUs on servers and NVIDIA Tesla T4 GPUs on rack nodes. Managed Service for Apache Spark allows specification of minimum CPU platforms like Intel Skylake across master and worker instances. Choosing between edge infrastructure, GPU clusters, and TPU pods requires weighing local networking boundaries, hardware ownership, and framework acceleration capabilities.
Tensor Processing Units (TPUs) are Google's custom-designed accelerators optimized for the large-scale matrix operations required in neural network training. A TPU v2 slice contains 512 chips connected by reconfigurable high-speed links, providing the bandwidth necessary for distributed training. The --accelerator-type flag in TPU creation commands specifies both the TPU version and core count—for example, v2-8 creates a single v2 TPU while v2-128 creates a slice with 128 TensorCores.
Google Cloud offers multiple TPU generations, each with different machine type prefixes: TPU v5e uses ct5lp-, v5p uses ct5p-, v4 uses ct4p-, and v3 uses ct3p- or ct3-. Topologies are configured as two or three-dimensional arrangements (such as 2x4, 4x4, or 8x16) that determine how chips are interconnected and communicate. Larger topologies provide more aggregate compute but require more complex communication coordination between devices.
GKE supports both single-host and multi-host TPU node pools. A single-host configuration places all TPU chips on one virtual machine, while multi-host configurations distribute chips across multiple VMs connected via high-speed interconnects. For instance, a TPU v6e (Trillium) with topology 8x16 spans 128 chips across 32 VMs. The machine type ct6e-standard-4t supports single-host mode with topology 2x2 (4 chips, 1 VM) or multi-host modes like 4x4 (16 chips across 4 VMs) or 8x16 (128 chips across 32 VMs).
Google Kubernetes Engine manages TPU resources through the Compute Engine API, automating provisioning and scaling of TPU node pools. GKE handles the underlying infrastructure complexity while traffic flows from user requests through GKE's load balancing to the TPU node pools where training jobs distribute data across available accelerators. GKE supports TPU Trillium (v6e), v5e, v5p, v4, and v3 versions across different regions and availability zones.
JetStream provides high-performance inference serving optimized for TPUs and GPUs. For distributed inference workloads, JetStream employs continuous batching to dynamically group incoming requests into batches, reducing latency and increasing throughput. Additional optimizations include KV cache quantization, which compresses the key-value cache used in attention mechanisms, and Int8 weight quantization, which reduces model weight precision from 32-bit floating point to 8-bit integers for faster computation.
Google Distributed Cloud connected provides dedicated on-premises hardware that supports GPU-based training workloads. Distributed Cloud connected racks support GPU workloads on legacy configurations, while connected servers support GPU workloads in three-machine configurations. This enables organizations to run distributed training with low latency to local data sources, though the hardware requires connectivity to Google Cloud for management and monitoring. The control plane runs locally, and workloads continue for up to seven days if cloud connectivity is temporarily lost.
Model parallelism partitions large models across multiple accelerators when a single device cannot hold the entire model. Two primary techniques exist: pipeline parallelism and tensor parallelism. Pipeline parallelism (used in frameworks like GPipe and PipeDream) splits model layers or stages across different accelerators, with training batches divided into micro-batches that flow through the pipeline sequentially. Tensor parallelism (as seen in Megatron-LM) splits individual tensors or weight matrices within layers across devices, with each device computing a portion of the operation. Pipeline parallelism suits models with sequential layers, while tensor parallelism works well for models with large, monolithic layers.
Pipeline parallelism uses micro-batches to keep the pipeline full and minimize idle time. The forward pass of one micro-batch and the backward pass of another can overlap on different devices to improve hardware utilization. However, this requires careful gradient synchronization—gradients must be accumulated across micro-batches and synchronized across devices only after the entire batch completes processing. This creates a trade-off between throughput (how fast the model trains) and latency (how quickly gradients update).
Pipeline bubbles occur when devices sit idle waiting for data from preceding stages, reducing training efficiency. Increasing the number of micro-batches reduces bubbles but increases latency for gradient updates. Communication overhead arises from transferring activations and gradients between devices, particularly in tensor parallelism where layer splits require frequent all-reduce operations. The optimal configuration balances model partition, micro-batch size, and device placement to minimize these inefficiencies for a given hardware topology.
Device placement determines where model partitions physically sit on accelerators, and this choice must consider the hardware interconnect topology. NVLink between GPUs or the high-speed toroidal mesh network within a TPU pod provides high-bandwidth, low-latency communication for frequent data transfers. On Google Cloud, this involves selecting appropriate machine types (like A2 or G2 VM instances with NVIDIA GPUs or TPU vX machines) and configuring workloads to leverage the underlying architecture. TPUs are particularly optimized for large-scale matrix operations with built-in fast interconnect, making certain tensor parallelism patterns more efficient.
Data parallelism replicates the entire model on each accelerator and splits the training data, requiring synchronization of gradients across all devices after each batch. Model parallelism partitions the model itself across multiple accelerators, with each device holding only a portion of the model—useful when models are too large to fit in a single device's memory.
Choose multi-host TPU slices when training very large models that require many TPU chips working together with high-bandwidth communication. Choose single-host configurations when the workload fits on fewer chips or when fault isolation matters, since a hardware failure in one multi-host slice affects all connected VMs.
JetStream reduces inference costs through Int8 weight quantization (which uses less memory by storing weights as 8-bit integers instead of 32-bit floats) and KV cache quantization (which compresses attention mechanism states). Continuous batching also improves throughput by grouping multiple requests together, making better use of available compute.
Professional Machine Learning Engineer
Prepare and test your skills
Prepare and test your skills
StudyCerts
StudyCerts