Professional Cloud Network Engineer
Google Kubernetes Engine (GKE) gives administrators the choice between public and private cluster nodes, and this choice affects how workloads connect to the internet and how secure the cluster is. Public nodes have external IP addresses and can communicate directly with the internet. Private nodes have only internal IP addresses and cannot be reached from the internet without special configuration. The decision between these two options impacts security, cost, connectivity to Google APIs, and how you manage network traffic for your applications.
Private cluster nodes are Compute Engine virtual machines that receive only internal IP addresses from the subnet's primary IP range and do not have external IP addresses. Public cluster nodes receive external IP addresses directly, which allows them to communicate with the internet if firewall rules permit it. In GKE Standard, you can configure private nodes at the cluster level or override settings at the individual node pool level. In GKE Autopilot, you configure privacy at the workload level instead.
Choosing private nodes improves security because workloads and virtual machines stay hidden from direct internet exposure. The GKE control plane still communicates with private worker nodes using internal IP addresses only. However, when you convert a public node pool to private, workloads lose their ability to reach the internet unless you set up translation or routing mechanisms like Cloud NAT.
Private Google Access (PGA) is a VPC feature that lets private nodes and Pods reach Google APIs and container registries without needing external IP addresses. Traffic using PGA stays inside Google's internal network instead of going across the public internet. When you create a private cluster, GKE automatically enables PGA on the subnet, except for clusters inside a Shared VPC where you must enable it manually.
Administrators can configure custom DNS zones to direct Google API traffic to specific internal IP ranges. Traffic can route to private.googleapis.com using the IP range 199.36.153.8/30 or to restricted.googleapis.com using 199.36.153.4/30 when using VPC Service Controls. Workloads authenticate securely using Workload Identity Federation for GKE, which connects Kubernetes service accounts directly to IAM permissions so Pods can query Google Cloud services without going through external egress gateways.
Cloud NAT provides outbound internet connectivity for private nodes and Pods without assigning them public IP addresses. When you configure Cloud NAT for a private GKE cluster, the gateway must cover both the primary subnet IP range used by nodes and the secondary IP ranges assigned to Pods and Services. In a Shared VPC environment, you must deploy the Cloud NAT gateway in the host project where the network lives.
GKE VPC-native clusters allocate alias IP ranges to nodes, so Cloud NAT allocates translation resources based on the underlying node VMs. In static port allocation, Cloud NAT reserves at least 1,024 ports per node VM. Dynamic port allocation adjusts between a configured minimum (such as 64 ports) and maximum (such as 2,048 ports) based on demand. To prevent port exhaustion during high connection churn, you can reduce the TCP TIME_WAIT timeout from 120 seconds to 5 seconds and enable the --disable-default-snat flag on GKE clusters.
Cloud Service Mesh provides standalone forward proxies called egress gateways to control, authorize, and observe outbound traffic leaving the cluster at the application layer. Traffic flows from the application container to an Envoy sidecar proxy, then across the internal network to a dedicated egress gateway proxy, and finally toward the external destination. This architecture separates application developer workflows from network security policies by enforcing Layer 7 authorization policies alongside Layer 4 transport controls.
To prevent workloads from bypassing egress controls, you should run egress gateways on a dedicated gateway node pool that is isolated from standard application workloads. You place taints on gateway nodes and add matching tolerations only to the egress gateway deployment, while using a node selector to direct proxy Pod placement. Restrictive VPC firewall rules applied to default node pools block direct outbound connections, while separate firewall rules permit egress traffic only from the dedicated gateway nodes.
Public cluster nodes assign ephemeral external IP addresses to node VMs, which creates challenges for traditional perimeter firewall policies. Because Pod and node IP addresses change during scaling and upgrades, tracking fixed IP identities on external firewall devices becomes difficult. To secure public nodes, you combine VPC firewall rules with Kubernetes network policies.
VPC firewall rules apply Layer 4 filtering to node VMs using service accounts or network tags, ensuring different node pools receive different network permissions. Inside the cluster, Kubernetes network policies or GKE Dataplane V2 enforce Layer 4 filtering at the Pod level using namespace and Pod labels to define allow and deny rules. A default-deny policy ensures newly deployed workloads cannot start unauthorized egress connections regardless of the node's public IP state.
Private nodes are GKE worker nodes that have only internal IP addresses and no direct inbound or outbound internet connectivity. This configuration enforces network isolation by removing a direct attack surface from the public internet. Workloads on these nodes cannot reach the internet unless you enable a NAT solution like Cloud NAT. The control plane communicates with all nodes using internal IPs only. This model supports zero-trust frameworks and strict regulatory compliance such as PCI DSS or GDPR by keeping sensitive data and workloads within a private network boundary.
Public nodes have external IP addresses, providing direct internet connectivity. This model requires strong security measures to manage the increased external exposure. You must implement strict VPC firewall rules and IAM policies to control inbound and outbound traffic. Without these additional layers, public nodes create a larger attack surface and make the cluster more vulnerable to external threats. This model works well for development environments or workloads that need low-latency direct internet egress.
The trade-off centers on connectivity versus isolation. Private nodes enhance security by default but add complexity for outbound traffic, requiring Cloud NAT for internet access and Private Google Access for Google APIs. Public nodes simplify connectivity but shift the security burden to perimeter controls. Private nodes can reduce costs by eliminating external IP charges for nodes that do not need them. However, converting a public node pool to private can break workloads if dependencies like Private Google Access or Cloud NAT are not set up first, especially in Shared VPC networks.
Choosing private nodes is a primary control for meeting compliance requirements that demand network isolation for sensitive workloads. To build defense-in-depth, you layer additional controls regardless of node type. This includes enabling GKE Dataplane V2 for built-in network policy enforcement and logging, applying Kubernetes network policies to restrict Pod-to-Pod communication, and using VPC Service Controls to create a service perimeter. For private clusters, you secure egress using dedicated egress gateways on hardened node pools and apply granular VPC firewall rules based on node service accounts or network tags.
A private GKE cluster has nodes with only internal IP addresses that are not connected to the internet by default. Private nodes cannot receive inbound traffic from the public internet, which significantly reduces the cluster's attack surface. You can enable private nodes at the cluster level or at the node pool level for Standard clusters, allowing you to mix public and private node pools within the same cluster. If you convert a public node pool to private, workloads that need external access may fail unless you configure Cloud NAT for outbound internet access or enable Private Google Access for Google APIs.
Public nodes have both internal and external IP addresses, allowing them to communicate with the internet directly. Public nodes work well for workloads that must be directly accessible from the internet or that need low-latency access to external services without NAT overhead. However, public nodes expose your workloads to potential external attacks, so you must weigh the accessibility benefit against the security risk for each workload.
You can associate different service accounts with different node pools, then configure VPC firewall rules to apply different Layer 4 controls to each node pool based on its service account or network tags. This approach lets you enforce stricter security on sensitive node pools while allowing more permissive access for workloads that need it. For example, you might allow external traffic only to a node pool running public-facing web servers while blocking all external traffic to a node pool running backend databases.
Kubernetes network policies provide additional isolation by controlling traffic at the Pod level rather than the node level. Network policies are scoped to namespaces and can be applied to specific Pods within a namespace, giving you finer-grained control over which Pods can communicate with each other and with external endpoints. When you apply a network policy to a namespace, all traffic that is not explicitly allowed by the policy is denied by default.
For clusters requiring controlled outbound access, you can deploy egress gateways on dedicated gateway node pools. The egress gateway acts as a single exit point for all traffic leaving the cluster, allowing consistent security policies, logging, and monitoring for outbound connections. You configure the egress gateway deployment with a node selector to ensure it runs only on nodes in the dedicated gateway node pool, and you use taints and tolerations to prevent other workloads from running on those nodes.
When using egress gateways, you should configure Cloud NAT to allow the gateway to make enough simultaneous connections to external destinations. Setting the minimum number of ports per VM appropriately helps avoid port exhaustion and connection reuse delays. Increasing the number of egress gateway replicas reduces the chances of endpoint-independent mapping conflicts, which occur when multiple connections compete for the same NAT mappings.
Private Google Access enables nodes and Pods on private clusters to access Google APIs and services using a set of four IP addresses without requiring external IP addresses. You can use either private.googleapis.com (199.36.153.8/30) or restricted.googleapis.com (199.36.153.4/30), depending on whether you are using VPC Service Controls. Traffic to these addresses never leaves the Google network, providing secure access to Google APIs without exposing workloads to the public internet.
Workload Identity is the recommended way to allow GKE workloads to authenticate with Google APIs. Instead of attaching service account keys to nodes or Pods, you bind a Kubernetes service account to a Google Cloud service account, following the principle of least privilege. When you use Workload Identity with Private Google Access, workload Pods can bypass the egress gateway and connect directly to Google APIs and services while maintaining security through IAM permissions.
Node pools support independent scaling through the cluster autoscaler, which dynamically adds and removes nodes based on workload demand. When planning for scaling, you must account for the maximum size of all node pools in your IP address planning, because each new node requires its own node IP address and its own allocatable set of Pod IP addresses. You can configure different Pods-per-node limits for different node pools, which affects IP address consumption and allows you to optimize allocation based on workload characteristics.
Node pool upgrades are managed separately from cluster control plane upgrades, letting you control when node software is updated. You can enable automatic node upgrades for each node pool, or you can manually upgrade nodes when needed. For node pools with specific security requirements such as gateway nodes, you may want to schedule upgrades during maintenance windows to minimize disruption to sensitive workloads.
Control plane endpoints define how administrators and services connect to and manage components like GKE clusters, ingress controllers, and service meshes. Choosing between public and private endpoints involves balancing network reachability with security isolation.
Private control plane reachability is the ability to connect to a Google Kubernetes Engine (GKE) cluster's management interface from networks outside its main VPC, such as an on-premises data center or a peered VPC. This is essential for securing management traffic. The GKE control plane's private endpoint is an internal IP address that is only reachable within its own VPC network. By default, it is not accessible from other networks connected via VPC Network Peering or from on-premises environments using Cloud VPN or Cloud Interconnect.
To enable direct access, you must advertise the route to the control plane's private IP address. You do this by configuring custom route advertisements on the Cloud Router for your hybrid connection, which uses BGP to tell your on-premises network how to reach the endpoint. For access from any Google Cloud region in a hybrid setup, you can enable the --enable-master-global-access flag when creating the cluster.
In a hub-and-spoke VPC architecture with peering, you cannot reach the private control plane directly from a spoke VPC or from on-premises via the hub. Since VPC peering is non-transitive, the spoke VPC can only talk to the hub VPC, not through it to the GKE cluster's network. To solve this, you must deploy a forward proxy, like a bastion host, inside the VPC that hosts the GKE cluster. All management traffic from the external network must flow first to this proxy, which then forwards it to the private control plane endpoint.
Master Authorized Networks is a GKE security feature that restricts access to the cluster's control plane by specifying a list of trusted source IP address ranges (CIDR blocks). This creates a zero-trust network boundary, allowing only traffic from approved networks. When you configure authorized networks, you list the specific IP subnets allowed to reach the GKE control plane. For a private endpoint, these should be private IP ranges. For a public endpoint, you can allow public or internal ranges. To reach a private endpoint from on-premises, you use custom route advertisements.
Choosing between a public or private control plane endpoint depends on your security needs. A private endpoint keeps the Kubernetes API server accessible only from within your VPC or authorized on-premises networks, removing it from the public internet and reducing the attack surface. Even if you disable the public endpoint for user access, Google still uses it for necessary cluster management like maintenance and upgrades. Using a private endpoint together with authorized networks provides defense-in-depth, requiring both network-level access and identity verification.
In a Shared VPC setup, network resources are centralized in a host project while GKE clusters run in service projects. The Google Kubernetes Engine Service Agent in each service project needs the roles/container.hostServiceAgentUser role on the host project. This role allows the agent to manage network resources. Cluster creation also requires the Compute Network User role so node instances can attach to the host subnet, using its primary IP range for nodes and designated secondary ranges for Pods and Services.
Gateway and ingress controllers can be configured with private or public addresses. The gke-l7-rilb GatewayClass assigns a regional private IP address from the node's IP range, keeping all ingress traffic within the private VPC. The gke-l7-regional-external-managed or gke-l7-global-external-managed GatewayClasses assign regional or global public IP addresses from Google's pools. Administrators can also assign static named IP addresses to maintain a consistent endpoint address.
Configuring secondary IP ranges for Google Kubernetes Engine (GKE) is essential for VPC-native clusters that need IP addresses for Pods and Services. A VPC-native cluster requires a subnet with a primary IP address range for nodes and internal load balancers, plus at least two secondary IP address ranges: one for Pod IP addresses and a separate one for Kubernetes Service addresses.
The primary IP address range supplies addresses for GKE nodes and internal load balancers, while the secondary ranges are dedicated to Pods and Services. For Pods, GKE allocates a block of alias IP addresses from the secondary range to each node, typically a /24 per node supporting up to 110 Pods. For Services, a separate secondary range provides stable virtual IP addresses called ClusterIPs. In GKE Standard clusters version 1.29+ and Autopilot clusters version 1.27+, Service addresses can alternatively be assigned from a Google-managed range, eliminating the need for a user-managed secondary range.
Effective IP address planning must account for the maximum number of nodes, the configured Pods per node, and the total number of Services. The size of the Pod secondary range determines the maximum number of nodes a cluster can support because each node reserves a contiguous block from that range. For example, a /14 Pod range with 262,144 addresses can support approximately 900 nodes with the default Pods-per-node setting. You should use a custom subnet mode rather than auto mode to explicitly define non-overlapping ranges. In a Shared VPC architecture, a network administrator in the host project must pre-create these subnet and secondary ranges before a service project administrator can deploy a cluster. These secondary ranges must not overlap with the primary range, other cloud services, or connected on-premises networks.
If RFC 1918 private address space is insufficient, you can use non-RFC 1918 private addresses like 100.64.0.0/10 or privately used public IP (PUPI) addresses for Pod secondary ranges. When using PUPIs or other non-RFC 1918 addresses, you must configure the cluster to handle Source Network Address Translation (SNAT) correctly. For Standard clusters, you typically need to either disable default SNAT at cluster creation or configure the IP masquerade agent to exclude the Pod and Service CIDRs from masquerading. This prevents traffic from these addresses from being incorrectly SNAT'd when exiting the node, which would break return routing. PUPI addresses require careful route management to prevent conflicts when connecting to other networks.
Designing primary IP ranges for subnets involves selecting appropriate CIDR blocks that accommodate current resource requirements while allowing for future growth without requiring subnet reconfiguration. The primary IP address range also determines the IP addresses available for internal load balancers and nodes within that subnet. When planning primary IP address ranges for GKE clusters, you must account for the number of nodes the cluster needs to support, plus additional addresses for internal load balancers. The first two and last two IP addresses of every primary IP address range are reserved by Google Cloud, reducing usable addresses by four.
Internal load balancers that use Envoy proxy architectures, including cross-region internal Application Load Balancers and cross-region internal proxy Network Load Balancers, require a dedicated proxy-only subnet in every region where backends reside. The proxy-only subnet provides a pool of IP addresses that Google Cloud uses to run managed Envoy proxies, which terminate incoming client connections and establish new connections to backend VMs. Proxy-only subnets must be configured with an active role and a specific purpose such as REGIONAL_MANAGED_PROXY or GLOBAL_MANAGED_PROXY. Google Cloud enforces strict allocation rules between proxy-only subnets and backend workloads: a proxy-only subnet is reserved exclusively for Google-managed Envoy proxies and cannot be assigned to VM instances, backend network endpoint groups, or forwarding rules.
IP address planning for Google Kubernetes Engine clusters involves selecting appropriate IP address ranges for nodes, Pods, and Services while ensuring no overlap with existing networks and sufficient space for cluster scaling. GKE supports multiple IP addressing options including RFC 1918 private addresses, non-RFC 1918 private addresses, and privately used public IP (PUPI) addresses, each with distinct characteristics that make them suitable for different scenarios.
RFC 1918 private IP address ranges (10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16) are the recommended and most common choice for GKE clusters. These addresses are not routable on the public internet, making them inherently secure for internal cluster communication. VPC-native clusters allocate node IP addresses from the primary subnet range, while Pods and Services receive addresses from secondary ranges within the same subnet. RFC 1918 addresses work seamlessly with VPC Network Peering, Cloud VPN, Cloud Interconnect, and Private Service Connect without requiring special configuration.
When organizations exhaust RFC 1918 address space, non-RFC 1918 private addresses provide additional flexibility. The RFC 6598 address space (100.64.0.0/10) is commonly used for this purpose and is recommended over Class E (240.0.0.0/4) addresses due to interoperability concerns with certain on-premises hardware and Windows hosts. Non-RFC 1918 addresses require the same VPC-native alias IP configuration as RFC 1918 addresses but demand careful planning to avoid conflicts with other private address spaces in hybrid environments. For Standard clusters using non-RFC 1918 addresses, administrators must either explicitly disable default SNAT or configure the IP masquerade agent to exclude cluster Pod IP addresses from SNAT processing.
Privately used public IP (PUPI) addresses are public IP address ranges that organizations use privately within their VPC networks without owning the addresses. PUPI addresses serve as an effective solution when RFC 1918 space is exhausted and non-RFC 1918 space is unavailable or already in use. PUPI addresses can only be applied to Pod IP address ranges, not to node or Service IP addresses. When using PUPI addresses, the IP masquerade agent must be configured to perform Source Network Address Translation (SNAT) on outbound traffic from Pods; otherwise, return traffic routes over the public internet and fails. For Autopilot clusters, SNAT for PUPI traffic requires no additional configuration, while Standard clusters require explicit SNAT enablement.
GKE clusters use specific Google-managed IP address ranges for cluster operations. The control plane in Private Service Connect clusters requires a /28 subnet within the RFC 1918 private ranges that does not overlap with any other CIDR in the VPC network. For Services, GKE Autopilot clusters running version 1.27 and later and Standard clusters running version 1.29 and later use the 34.118.224.0/20 IPv4 address range by default, which Google manages internally without publishing routes on the public internet.
Private Service Connect (PSC) enables private connectivity between VPC networks and Google Cloud services without exposing traffic to the public internet. When using PSC with GKE, the primary subnet range provisions the internal IP address assigned to the control plane endpoint, though this can be overridden using the private-endpoint-subnetwork flag. PSC also allows organizations to reduce IP address consumption by providing connectivity between VPCs with overlapping IP addresses, making it valuable for multi-tenant architectures or scenarios where IP address space is severely constrained. However, PSC endpoints can only be reached from the same region as the service and cannot be accessed from peered VPCs or networks connected through Cloud VPN or Cloud Interconnect.
Shared IP range strategies consolidate address consumption by reusing secondary ranges or sharing single virtual IP addresses among multiple network services. In large environments, network teams can configure multiple GKE clusters to share the same secondary subnet ranges for Pods and Services to minimize overall address consumption. While sharing secondary Pod ranges across clusters reduces subnet proliferation, traffic management requires strict planning because a single expanding cluster can consume the entire available allocation. To optimize internal load balancing resources, GKE allows internal TCP/UDP load balancers to share a single virtual IP address across up to 50 distinct backend services using different port configurations.
This section covers the design considerations and implementation steps for enabling IPv6 in Google Kubernetes Engine (GKE) clusters.
This section covers the architectural patterns and configuration options for implementing various load balancer types to distribute traffic to applications running on Google Kubernetes Engine (GKE).
This section covers the process and best practices for creating, modifying, and managing node pools within a Google Kubernetes Engine (GKE) cluster to support diverse workload requirements.
When you convert a public node pool to private, workloads lose external connectivity unless you first configure Cloud NAT for outbound internet access and enable Private Google Access for Google APIs. In Shared VPC environments, you must manually enable Private Google Access on the host project. Workloads that depend on external services will fail until these connectivity mechanisms are in place.
You make it reachable by configuring custom route advertisements on the Cloud Router for your Cloud VPN or Cloud Interconnect connection. This advertises the specific IP route of the private endpoint into your on-premises routing table via BGP, allowing traffic to flow from on-premises to the control plane.
The primary IP address range provides IP addresses for VM instances, nodes, and internal load balancers. Secondary IP address ranges provide additional IP addresses for specialized purposes such as GKE Pods, Kubernetes Services, and alias IP addresses for containers. Primary ranges cannot be changed after subnet creation, while secondary ranges can be added or modified.
Use PUPI addresses when your organization has exhausted RFC 1918 address space and non-RFC 1918 space is unavailable or already in use. PUPI addresses can only be applied to Pod IP ranges, not to nodes or Services, and require the IP masquerade agent to be configured for SNAT on Standard clusters.
Prepare and test your skills
Prepare and test your skills
StudyCerts
StudyCerts