Professional Cloud Network Engineer
Cloud NAT is a distributed, software-defined managed service that provides outbound network address translation (NAT) for Google Cloud workloads without requiring public IP addresses on individual virtual machines. In a Google Kubernetes Engine (GKE) environment, VPC-native clusters assign node virtual machines primary IP addresses and allocate secondary subnet IP address ranges to Pods as alias IP ranges. Configuring Cloud NAT for both primary and secondary subnet ranges ensures that nodes, workloads within Pods, and cluster services can initiate outbound internet connections while preventing unsolicited inbound traffic from external networks.
A Public NAT gateway translates outbound internal IPv4 packets from private GKE clusters to external IPv4 addresses on the internet. To enable full outbound internet access for an entire VPC-native cluster, the Cloud NAT gateway must be configured to apply to all IP address ranges in the cluster's subnet, including the primary range used by nodes and the secondary ranges used for Pods and Services. If a gateway is configured only for the primary range, node hosts can connect to the internet, but packets originating directly from secondary Pod IP ranges are dropped. Cloud NAT policies apply at the subnet range level, meaning network administrators cannot selectively enable or disable NAT for individual Pods within the same secondary IP range.
Outbound translation for GKE Pod traffic depends on the relationship between node-level IP masquerading and VPC-level Cloud NAT. By default, the in-cluster IP masquerade agent performs source network address translation (SNAT) on the node, rewriting the source IP address of outbound Pod packets to match the node's primary internal IP address before the traffic leaves the virtual machine. When this traffic exits the node, Cloud NAT performs a second translation step, mapping the node's internal IP address to an external public NAT IP address. Administrators can prevent this double SNAT behavior by enabling the --disable-default-snat flag during cluster creation or by adding destinations to the nonMasqueradeCIDRs configuration list. Disabling default in-cluster SNAT offloads all translation tracking to Cloud NAT and preserves Pod IP address visibility in VPC Flow Logs and Packet Mirroring until traffic reaches the network boundary.
Cloud NAT manages connection scalability by reserving source IP addresses and transport layer source ports for each GKE node virtual machine (VM). In VPC-native clusters, each node receives an alias IP netmask smaller than /32, which allows the node's allocated NAT source ports to be shared across all running Pods on that host. Administrators can configure port assignment using static or dynamic models:
To prevent dropped packets during traffic spikes, administrators can assign additional static public IP addresses to the gateway or increase the maximum port limit per VM. Lowering the TCP TIME_WAIT timeout setting from its default of 120 seconds to 5 seconds also accelerates port reuse when workloads generate frequent short-lived connections to the same destination endpoints.
Configuring a Public NAT gateway automatically activates Private Google Access across all associated primary and secondary subnet ranges. When Pods or nodes send requests to Google APIs and services using default domains or private endpoints such as private.googleapis.com (199.36.153.8/30) or restricted.googleapis.com (199.36.153.4/30), traffic bypasses the public NAT IP translation and stays entirely within Google's internal network. Private Google Access remains enforced on the subnet as long as the Cloud NAT gateway serves that range and cannot be manually disabled. This integration ensures that container image pulls and API requests proceed securely over internal routes without consuming Cloud NAT port allocations or public IP capacity.
The GKE IP Masquerade Agent is an in-cluster component that controls whether a Pod's source IP address is translated to the node's primary internal IP address when traffic leaves the cluster. This process, known as IP masquerading or node-level SNAT, is the default mechanism for traffic destined outside the cluster's Pod IP range. The agent's configuration determines which destination IP ranges preserve the original Pod IP address, maintaining accurate source identity for network security policies, logging, and hybrid network routing.
By default, GKE automatically creates an EgressNATPolicy named default that specifies destination CIDR ranges where traffic is not masqueraded. This default policy applies the NoSNAT action to standard private IP ranges (RFC 1918: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) as well as Carrier-Grade NAT ranges (100.64.0.0/10). Outbound traffic sent to any IP address inside these CIDR blocks retains the original Pod IP address as the packet source, which is required when connecting across peered VPC networks or hybrid connections like Cloud VPN and Cloud Interconnect. For any destination outside these non-masquerade ranges, such as the public internet, the node rewrites the packet source to the node's internal IP address.
Administrators can modify the default non-masquerade behavior by editing the existing default EgressNATPolicy manifest. Executing kubectl edit egressnatpolicies default opens the configuration file, allowing administrators to add or remove CIDR blocks under the spec.destinations list using the NoSNAT action. When a new CIDR range (such as 192.0.2.0/24) is added, outbound Pod traffic directed to that range bypasses node masquerading and preserves the Pod IP. Changes made to the default policy are processed by the GKE control plane and take up to three minutes to propagate to the ip-masq-agent DaemonSet running on every node.
Deploying custom EgressNATPolicy resources alongside the default policy allows administrators to manage non-masquerade rules in a modular, additive structure. An administrator creates a YAML manifest with a unique policy name and defines a list of destination CIDR ranges marked with the NoSNAT action, then applies it using kubectl apply -f. This approach allows specific exceptions to be added for partner networks or external services without altering the base default configuration. Multiple policies coexist simultaneously in the cluster, and the local agent reconciles all active policy definitions into a unified set of translation rules.
Verifying IP masquerade configurations involves checking custom resource definitions (CRDs), active policies, and cluster synchronization events. Administrators run kubectl get crds to confirm CRD registration, inspect active rules with kubectl get egressnatpolicies, and monitor status logs with kubectl get events. During operation, the ip-masq-agent on each node continuously monitors these policy resources to maintain the node's internal routing tables. When a Pod sends an outbound packet, the node inspects the destination IP against all configured NoSNAT CIDRs; matching destinations preserve the Pod IP, while non-matching destinations are translated to the node's primary internal IP.
Cloud NAT provides outbound address translation for Google Cloud workloads, mapping internal VM and Pod alias IP addresses to public external IP addresses. In GKE VPC-native clusters, Pod IP addresses reside within alias IP ranges attached to node VMs, which allows Cloud NAT to allocate external source IPs and port blocks at the VM level. When high connection volumes exhaust allocated source ports or gateway capacity, outbound connections fail, requiring administrators to diagnose translation errors and adjust allocation settings.
Packet loss occurs in Cloud NAT when a VM exhausts its assigned pool of external source ports and IP addresses, preventing new outbound connections from being translated. Administrators can identify dropped packets in Cloud Logging by querying the NAT gateway resource, region, and filtering for an allocation status of DROPPED. While logs show when packets are dropped, they do not identify the root failure condition on their own. For deeper troubleshooting, Cloud Monitoring categorizes drops under specific error codes: OUT_OF_RESOURCES for exhausted ports or IPs, ENDPOINT_INDEPENDENT_CONFLICT for mapping collisions, and NAT_ALLOCATION_FAILED for gateway IP allocation limits.
Monitoring translation health requires querying specific Cloud NAT metrics in Cloud Monitoring to track drop rates and resource limits over time. The primary metric for tracking failed connections is router.googleapis.com/nat/dropped_sent_packets_count, which should be filtered by gateway name, aligned to a rate per minute, and grouped by the drop reason. If drops occur with the NAT_ALLOCATION_FAILED reason, the Cloud NAT gateway has run out of available public IP addresses and requires additional static IPs or auto-allocation expansion. Reviewing gateway logs in parallel helps administrators determine if VM-level port limits need adjustment.
Choosing between static and dynamic port allocation directly controls how Cloud NAT distributes transport layer ports and responds to traffic spikes. Under static port allocation, each VM receives a fixed number of source ports (defaulting to a minimum of 1,024 ports per node) regardless of actual usage, which can lead to port exhaustion on busy nodes while leaving ports unused on idle nodes. Dynamic port allocation assigns a low starting threshold (such as 64 ports per VM) and scales up to a higher maximum (such as 2,048 ports per VM) based on active demand. For GKE environments, dynamic port allocation with a minimum of 64 ports and a maximum of 2,048 ports per VM provides efficient scaling across variable workloads.
Reducing NAT packet loss requires configuring both Cloud NAT gateway settings and application-level connection patterns. On the gateway, administrators can increase the maximum ports per VM under dynamic allocation, raise static port limits, or reduce the TCP TIME_WAIT timeout from 120 seconds to 5 seconds to recycle ports faster. At the application layer, workloads can reduce port consumption by implementing HTTP/2 multiplexing, establishing persistent connections, and using connection pooling. These application practices prevent rapid port consumption when making frequent requests to identical destination endpoints.
Double SNAT occurs when outbound Pod traffic is translated twice: first by the in-cluster IP masquerade agent to the node's internal IP, and second by Cloud NAT to a public IP. This double translation consumes extra processing resources on the node and hides the Pod's alias IP address from VPC Flow Logs and Packet Mirroring. Administrators avoid double SNAT by using the --disable-default-snat flag during cluster creation or configuring nonMasqueradeCIDRs, offloading all translation tracking to Cloud NAT across primary and secondary ranges. Retaining default in-cluster SNAT is only recommended when Pod IP addresses must be intentionally hidden behind node IPs across Cloud VPN or Cloud Interconnect routes to on-premises networks.
StudyCerts
StudyCertsEgressNATPolicy custom resources to apply the NoSNAT action to specified CIDR blocks, preserving Pod source IP addresses for internal and hybrid traffic.--disable-default-snat flag offloads translation tracking to Cloud NAT and preserves Pod IP visibility in VPC Flow Logs until packets reach the network edge.OUT_OF_RESOURCES, ENDPOINT_INDEPENDENT_CONFLICT, or NAT_ALLOCATION_FAILED errors when a virtual machine exhausts its allocated source ports or public IP addresses.TIME_WAIT timeout from 120 seconds to 5 seconds accelerates port reuse for workloads that generate high rates of short-lived outbound connections.Double SNAT adds processing overhead on the node and masks the Pod's alias IP address behind the node's primary IP, obscuring workload identity in VPC Flow Logs and Packet Mirroring. Disabling in-cluster SNAT offloads address translation entirely to Cloud NAT and preserves Pod-level visibility across the VPC network.
Node virtual machines will be able to initiate outbound internet connections, but packets originating directly from secondary IP ranges used by Pods and Services will be dropped at the Cloud NAT gateway. The gateway must be configured to cover all primary and secondary subnet ranges to provide internet access for container workloads.
A NAT_ALLOCATION_FAILED error indicates that the Cloud NAT gateway has exhausted its pool of external IP addresses. Administrators must assign additional static public IP addresses to the gateway or enable automatic IP address allocation to expand connection capacity.
Prepare and test your skills
Prepare and test your skills