Design, implement, and secure Google Cloud networks, from VPCs and hybrid connectivity to load balancing and Cloud Armor, and track your readiness for the Professional Cloud Network Engineer exam.
Prepare and test your skills
Prepare and test your skills
Path MTU Discovery (PMTUD) is a mechanism that allows network devices to discover the maximum packet size that can traverse a path without fragmentation. When a packet with the Don't Fragment (DF) bit set is too large for a network segment along the path, Google Cloud sends a Fragmentation Required or Packet Too Big ICMP message back to the sending instance, indicating the largest packet size that can be successfully transmitted. The sending instance's operating system then uses this information to adjust its packet size for subsequent transmissions, allowing the connection to continue without manual intervention.
PMTUD relies on specific firewall and forwarding configurations to function correctly across hybrid connections. VPC firewall rules must allow ICMP for IPv4 or ICMPv6 for IPv6 traffic from sources that match the original packet destinations; to simplify this, allowing ICMP and ICMPv6 from all sources is a common practice. Additionally, forwarding rules for internal passthrough Network Load Balancers and internal protocol forwarding must use the L3_DEFAULT protocol so they can process both the ICMP messages used for PMTUD and the protocol of the original packet being transmitted.
Fragmentation introduces performance overhead because routers must break larger packets into smaller segments, consuming CPU resources at each hop along the path. When PMTUD fails due to blocked ICMP messages, packets may be silently dropped—a condition known as blackholing—where the sender never learns that its packets are too large and cannot reach the destination. This makes proper firewall configuration critical for hybrid connections using Cloud Interconnect or HA VPN, where the network path crosses multiple administrative boundaries.
VPC networks support MTU values ranging from 1300 to 8896 bytes, with a default of 1460 bytes. For workloads using Cloud Interconnect or HA VPN, the MTU must account for the entire path between the on-premises network and Google Cloud, including any intermediate networks that may have lower MTU limits. When configuring MTU for hybrid connections, you should verify that all network segments along the path support the selected MTU value to avoid fragmentation or blackholing issues.
The Cloud VPN gateway has a fixed MTU size of 1460 bytes. This size accommodates the standard Ethernet MTU of 1500 bytes minus the 40 bytes of overhead added by the IPsec tunnel encapsulation (IP header, ESP header, and optional authentication trailer). For traffic to flow successfully from an on-premises network through a Cloud VPN tunnel to a Google Cloud Virtual Private Cloud (VPC), the peer VPN gateway on the on-premises side must be configured to support this same 1460-byte MTU. If the on-premises gateway or any intermediate network device uses a smaller MTU, packets exceeding that size will be fragmented or dropped, degrading performance. Therefore, validating and configuring consistent MTU settings across the entire path is a prerequisite for a functional hybrid connection.
In resilient architectures using HA VPN or Cloud Interconnect, MTU configuration must be validated not just for a single active path but for all potential failover paths. For HA VPN with active/active or active/passive tunnels, the same MTU (1460 bytes) applies to all tunnels in a gateway. When designing for failover, you must ensure the peer gateway's configuration for the backup tunnel also supports the correct MTU, so traffic continues without fragmentation after a failover event. A common pitfall occurs when multiple HA VPN gateways are configured in an active/passive mode; passive tunnels are not used for failover until all active tunnels on all gateways have failed, which can create unexpected bandwidth loss and complicate MTU validation across the broader topology.
For Cloud Interconnect (Dedicated or Partner), the MTU is typically 1500 bytes for the underlying physical connection, but the effective MTU for workloads must consider any additional encapsulation used in the network design, such as VLAN tags. The path through the interconnect and into the VPC must not introduce a lower MTU bottleneck. When using Cross-Site Interconnect for Layer 2 connectivity between on-premises sites over Google's network, the MTU is configurable and supports Jumbo frames (up to 9000 bytes) in VLAN mode, but this must be uniformly configured on all connected sites and matched to the capabilities of the attached on-premises routers.
Validation of MTU consistency is performed using diagnostic tools like ping with the Don't Fragment (DF) bit set and traceroute. You send ICMP echo requests with increasing packet sizes and the DF bit set from a source instance in Google Cloud to a destination in the on-premises network (or vice-versa). If a packet exceeds the MTU of any hop along the path, an ICMP "Fragmentation Needed" message should be returned, identifying the limiting MTU. Continuous validation should be performed across all redundant paths, including passive VPN tunnels or secondary Cloud Interconnect attachments, to ensure failover does not introduce MTU-related issues. This testing confirms the real-world data plane matches the designed configuration and helps avoid performance degradation during normal operation or failover scenarios.
The maximum transmission unit (MTU) defines the size, in bytes, of the largest packet that a network protocol layer can transmit, covering both packet headers and payload data. When network traffic traverses hybrid networking paths such as Cloud VPN or Cloud Interconnect, encapsulation protocols append additional header bytes to each packet. Calculating the effective MTU requires subtracting this encapsulation overhead from the underlying path MTU to prevent packet fragmentation, latency spikes, or dropped traffic. Properly aligning MTU configurations across virtual machine (VM) interfaces, Cloud Router instances, and external gateways ensures reliable and high-performance hybrid communication.
Encapsulation adds extra protocol headers to network packets, which reduces the total space remaining for user payload data. For workloads communicating across hybrid connections without encapsulation, systems standardly operate using a 1500-byte MTU setting across on-premises endpoints, internet paths, and private cloud interfaces. When traffic requires tunneling through VPN protocols such as IPsec or GRE, the effective MTU must be reduced based on specific endpoint configurations, generally resulting in an interface MTU between 1350 and 1390 bytes or lower. For environments utilizing VMware HCX, the default MTU setting on the HCX uplink profile is 1440 bytes.
Traffic type also dictates how MTU adjustments must be handled to protect application performance. While TCP-based traffic flows generally resolve fragmentation issues through Maximum Segment Size (MSS) clamping, large-packet UDP traffic cannot rely on MSS negotiation. For fragmentation-sensitive UDP traffic traversing public interfaces or internet paths, an MTU setting of 1370 bytes or lower prevents packet drops. Applying calculated MTU values directly to workload network interfaces prevents gateways from performing costly packet fragmentation along the hybrid network path.
A Virtual Private Cloud (VPC) network supports configurable MTU values ranging from the standard default of 1460 bytes up to a maximum allowed value of 8896 bytes. Increasing the VPC MTU to 8896 bytes enables jumbo frames, which reduces header overhead and increases payload data throughput by up to 10% compared to the 1460-byte default. Standard private clouds support internal endpoint MTU settings up to 8800 bytes, whereas stretched private clouds support settings up to 8600 bytes. Dedicated GPU instances—such as A2, G2, and A4 series—recommend an 8896-byte MTU, while specialized machine types like A3 with header split enabled require an 8244-byte MTU.
To use jumbo frame MTU sizes exceeding standard limits on Google Cloud VMs, the guest operating system must deploy the Google Virtual NIC (gVNIC) driver version 1.3 or later. Operating systems without native jumbo frame support can install updated gVNIC drivers manually to access high-bandwidth networking tiers. Verifying the VPC network MTU ensures that VM interfaces do not attempt to transmit packet frames that exceed the underlying network's physical layer capabilities.
Consistent MTU configuration across VLAN attachment resources and hybrid gateways prevents asymmetrical packet handling and unexpected data drops. When establishing physical or partner connections through Cloud Interconnect, network architects must configure the exact same MTU setting across all active VLAN attachments. For Cloud VPN deployments, cryptographic negotiations directly influence packet overhead and the resulting effective MTU. Because cipher negotiation can change dynamically during Security Association (SA) rekeying, Cloud VPN tunnels and peer VPN gateways must be configured to propose and accept only one stable cipher algorithm per cipher role.
Path MTU Discovery (PMTUD) requires Virtual Private Cloud (VPC) firewall rules to allow ICMP for IPv4 or ICMPv6 for IPv6 traffic from sources matching the original packet destinations. Additionally, forwarding rules for internal passthrough Network Load Balancers and internal protocol forwarding must use the L3_DEFAULT protocol to process both PMTUD ICMP messages and original packet traffic. If these configurations are missing and ICMP messages are blocked, PMTUD fails and causes silent packet dropping known as blackholing.
The Cloud VPN gateway uses a fixed MTU of 1460 bytes to accommodate the standard 1500-byte Ethernet MTU minus 40 bytes of overhead added by IPsec tunnel encapsulation. This 40-byte encapsulation overhead includes the IP header, ESP header, and optional authentication trailer. To avoid packet fragmentation or drops, peer on-premises gateways and all failover tunnels must be configured to match this 1460-byte MTU.
Enabling jumbo frame MTU sizes up to 8896 bytes requires configuring the Virtual Private Cloud (VPC) network MTU and deploying the Google Virtual NIC (gVNIC) driver version 1.3 or later in the guest operating system. Using an 8896-byte MTU reduces header overhead and increases payload throughput by up to 10 percent compared to the default 1460-byte setting. All connected network segments, including on-premises routers and interconnect attachments, must also support the selected MTU to prevent packet drops.
Administrators validate MTU consistency by running diagnostic tools such as ping with the Don't Fragment (DF) bit set and traceroute with incrementally larger packet sizes between Google Cloud and on-premises endpoints. If a packet exceeds the MTU of any hop along the path, an ICMP Fragmentation Needed message is returned to identify the bottleneck. This testing must be performed across all primary and redundant paths, including passive VPN tunnels and secondary interconnect attachments, to ensure failovers do not cause unexpected packet fragmentation.
Last updated