Professional Cloud Network Engineer

Provision Node Pools with Custom and Dedicated Pod CIDR Blocks

Provisioning node pools with custom and dedicated Pod CIDR blocks allows Google Kubernetes Engine (GKE) clusters to scale beyond a single subnet's IP limits and to separate network traffic for different workloads. This is achieved by connecting node pools and their Pods to multiple Virtual Private Cloud (VPC) networks and custom subnets, rather than having all Pods share one cluster-wide address range. Workloads then dynamically receive IP addresses from these dedicated secondary ranges or distinct VPC subnets.

Multi-Network Architecture and Dedicated Subnet Allocations

Multi-network support in GKE lets a single cluster attach node pools and Pods to multiple, isolated VPC networks, each with its own custom subnet CIDR block. An administrator first creates these custom VPC networks, configuring them with specific properties like a maximum transmission unit (MTU) size (e.g., 8244), and defines dedicated subnets with unique IP ranges for each virtual network interface. Ingress firewall rules are then applied to each custom network to allow necessary internal traffic, such as TCP, UDP, and ICMP, from designated source ranges. When a node pool is deployed into this environment, its virtual network interface cards (NICs) are mapped directly to these separate VPC subnets, creating isolated network paths for Pods.

Referencing Pod Networks and Custom Interface Configurations

Pods connect to these dedicated networks by referencing predefined Network resources in their YAML manifests. An administrator creates a Layer 3 or device-based Network object and specifies custom routes for target destination CIDRs. The Pod's specification then declares its primary interface using the networking.gke.io/default-interface annotation set to eth0 and lists additional network attachments in the networking.gke.io/interfaces annotation. When GKE schedules the Pod, the control plane assigns the primary IP address from the cluster's default subnet and secondary IP addresses from the custom networks, mapped to interfaces eth1 through eth8.

Multi-Network Policy Enforcement and Network Boundary Controls

Multi-network network policies enforce security by applying Pod-level firewall rules specifically to traffic moving across dedicated secondary and custom Pod networks. These policies require GKE Dataplane V2 and evaluate traffic flowing between Pods and Services over designated network attachments. A security policy must explicitly name the specific Pod network used by the target workload to filter its ingress and egress communication. Broader policies, like those using fully qualified domain names (FQDNs) or cluster-wide Cilium rules, are restricted because they affect all attached interfaces instead of isolating traffic on individual custom networks.

Network Driver Considerations and Disjoint Allocations

Workloads using advanced accelerators or requiring high-performance networking need clear separation between standard Pod ranges and dedicated hardware interfaces. The GKE DRANET driver manages specialized interfaces like those for remote direct memory access (RDMA), but it cannot operate on interfaces already configured with a secondaryPodRange. If a cluster uses disjoint secondary CIDRs alongside these specialized network profiles, administrators must ensure device management does not conflict with multi-network APIs on the same physical interfaces. Workloads claim these specialized devices through a ResourceClaimTemplate, guaranteeing dedicated throughput on specific network interfaces while keeping general Pod network routing isolated.

Manage Routing, Masquerading, and Security Across Disparate Pod Ranges

Managing routing, masquerading, and security for additional Pod ranges ensures seamless connectivity between Pods, Virtual Private Cloud (VPC) endpoints, and on-premises networks while conserving IP address space. This is critical when deploying clusters in separate VPC networks that use the same Pod IP address ranges, as it prevents routing conflicts and controls traffic visibility.

Configuring IP Masquerade Agent for Additional Pod Ranges

The IP masquerade agent is a GKE component that decides whether a node's outbound traffic keeps its original Pod IP address or is translated (masqueraded) to the node's own IP address. By default, it does not masquerade traffic destined for internal RFC 1918 ranges, keeping Pod IPs visible in logs. To manage traffic from additional Pod ranges—especially when those ranges are reused across clusters—you must explicitly configure the agent's nonMasqueradeCIDRs list. You add the cluster's node IP range and all Pod IP ranges to this list. This forces GKE to masquerade all traffic leaving the cluster, even to other Google Cloud resources, hiding the Pod IPs from the wider network.

Managing Route Advertisements with Cloud Routers

When GKE clusters in separate VPC networks share the same Pod IP ranges, you must prevent those ranges from being advertised between networks to avoid routing loops. You control this using Cloud Router custom route advertisements. After connecting VPCs via Cloud VPN tunnels or VPC Network Peering, you configure the Cloud Routers to stop advertising the shared Pod and Services IP address ranges. Conversely, you must manually advertise the distinct node IP ranges to connected on-premises networks. This creates an island-mode network where Pods are not directly reachable from outside their own cluster, while on-premises systems can still route traffic to the cluster nodes.

Implementing Security with VPC Firewall Policies

Security for hidden Pod ranges is simplified because the Pod IP addresses are not directly reachable across network boundaries. Since external traffic can only reach cluster Services exposed by internal load balancers or defined gateways, you need fewer explicit VPC firewall rules to block Pod-to-Pod communication across clusters. The primary security boundary shifts to controlling access to these exposed services. You apply VPC firewall policies at the network or organizational level to regulate traffic flowing through Cloud VPN tunnels or peering connections to the node IPs and load balancer IPs, rather than managing rules for every individual Pod IP.

Architectural Tradeoffs of Hiding Pod IP Ranges

Using separate VPC networks with hidden Pod IP ranges via Cloud VPN involves clear tradeoffs. The main advantage is the ability to reuse Pod and Service IP addresses across many clusters, conserving scarce RFC 1918 address space. A secondary benefit is simplified firewall management, as you don't need rules to block direct Pod access. The downsides include costs for Cloud VPN tunnels and data transfer, added complexity in synchronizing route advertisements during cluster creation, and bandwidth limits per VPN tunnel. You also lose the ability to apply VPC firewall rules or collect telemetry like VPC Flow Logs at the individual Pod level from central or peered networks, since all external traffic appears to come from node IPs.

Design and Allocate Secondary Subnet Ranges for Multi-Pod Architectures

Designing and allocating secondary subnet ranges involves planning additional Pod IPv4 address pools for VPC-native GKE clusters to prevent IP address exhaustion while maintaining network connectivity. In VPC-native clusters, Pod IPs are configured as secondary IP addresses (alias IPs) on each node's network interface, making them directly routable within the VPC without needing custom routes.

Secondary IP Address Ranges in VPC-Native Clusters

In a VPC-native cluster, Pod IPv4 addresses come from subnet secondary IPv4 address ranges, which are separate from the primary range used for nodes. Each cluster requires three IP ranges: a primary range for nodes (default /20), a secondary range for Pods (default /14), and a secondary range for Services (default /20). When you create a cluster, you assign a default Pod range, and for both Autopilot and Standard clusters, you can later configure additional Pod ranges that GKE will use for Pods on nodes in future node pools.

Adding Additional Pod IPv4 Ranges

You can expand the available IP space for an existing VPC-native cluster by adding more Pod IPv4 address ranges using discontiguous multi-Pod CIDR, without re-creating the cluster or configuring new subnets. The IPv4 addresses for these Pod ranges must come from a valid subnet range, including private RFC 1918 addresses (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) or privately used public IP addresses. For Standard clusters, you can also create node pools that each use a custom subnet secondary range for their Pod IPs, allowing different node pools to draw from different secondary ranges within the same subnet.

Cluster Sizing and IP Exhaustion Constraints

The maximum number of Pods a cluster can support depends on the size of the Pod subnet and the maximum Pods per node. The default is 110 Pods per node using a /24 range (256 addresses) reserved per node. By default, a cluster is limited to 1,024 nodes due to Pod IP address limitations. You can configure a Standard cluster to support up to 256 Pods per node by reserving a /23 range for each node. If you don't expect to run more than 64 Pods per node, you should reduce the maximum Pods per node setting to preserve IP address space, as a lower maximum lets you support more nodes on a fixed secondary IP range.

Configuring Node Pool Custom Pod Ranges

For Standard clusters, you can specify custom Pod IPv4 address ranges at the node pool level, assigning specific secondary ranges to particular node pools instead of using the cluster-wide default. These node pool custom ranges can be GKE-managed, where GKE automatically allocates the range, or user-managed, where you specify an existing secondary IP address range in the subnet. In Shared VPC environments, a Network Admin must create the subnet and secondary ranges before cluster creation, as GKE cannot manage secondary ranges in those configurations.

IP Address Utilization Monitoring

GKE calculates IP address utilization using data from Network Analyzer insights and other sources, available for all VPC-native clusters. You can view this on the cluster's Details page in the Google Cloud console under the Observability or Networking sections. The Status of IP allocation column shows the percentage of allocated IP addresses in a Pod's IP range. GKE considers all IP addresses in a node's assigned CIDR range as allocated, whether individual IPs are assigned to Pods or not, so the percentage reflects the entire Pod range reserved for a node, not just the IPs in active use.

Solutions for IP Exhaustion

When IP address utilization is high (approaching 100%), you have several solutions. You can add more Pod IPv4 address ranges using discontiguous multi-Pod CIDR, add more subnets with new IPv4 ranges so new node pools use IPs from these new subnets, or create a new cluster with a lower maximum Pods per node value to reserve fewer IPs per node. If you lack enough RFC 1918 address space, you can consider non-RFC 1918 ranges like the RFC 6598 space (100.64.0.0/10) or Class E addresses, though Class E addresses can cause interoperability issues with on-premises hardware and may require configuring the IP masquerade agent to exclude Pod IPs from source network address translation (SNAT).

Key Facts

• Multi-network GKE architectures allow node pools and Pods to attach to multiple, isolated VPC networks with custom subnet CIDR blocks, enabling scale and traffic separation.
• The IP masquerade agent must be configured to add the cluster's node and Pod IP ranges to its nonMasqueradeCIDRs list to hide Pod IPs when using additional ranges across separate VPCs.
• In VPC-native clusters, Pod IP addresses are allocated from subnet secondary IPv4 address ranges, distinct from the primary range used for nodes.
• You can add more Pod IPv4 ranges to an existing VPC-native cluster using discontiguous multi-Pod CIDR without re-creating the cluster.
• For Standard clusters, node pools can be configured with custom Pod IPv4 ranges, which can be either GKE-managed or user-managed from existing subnet secondary ranges.

Frequently Asked Questions

What is the purpose of configuring the IP masquerade agent for additional Pod ranges?

Configuring the IP masquerade agent forces all outbound traffic from the cluster to be translated to the node's IP address, hiding the original Pod IPs. This is essential when Pod IP ranges are reused across separate VPC networks to prevent routing conflicts and simplify security, as external systems only see the node IPs.

How does using separate VPC networks with hidden Pod IPs simplify firewall management?

When Pod IP ranges are hidden via masquerading and not advertised, external traffic cannot reach individual Pods directly. This shifts the security boundary to the cluster's exposed Services and node IPs, meaning you need fewer VPC firewall rules because you don't have to manage access controls for every Pod IP across network boundaries.

What is the difference between GKE-managed and user-managed custom Pod ranges for node pools?

GKE-managed custom Pod ranges are automatically allocated by GKE when you create the node pool. User-managed custom Pod ranges require you to specify an existing secondary IP address range that already exists in the subnet. In Shared VPC environments, only user-managed ranges are possible, as GKE cannot create secondary ranges in a shared network.