Path MTU Discovery (PMTUD) is a mechanism that allows network devices to discover the maximum packet size that can traverse a path without fragmentation. When a packet with the Don't Fragment (DF) bit set is too large for a network segment along the path, Google Cloud sends a Fragmentation Required or Packet Too Big ICMP message back to the sending instance, indicating the largest packet size that can be successfully transmitted. The sending instance's operating system then uses this information to adjust its packet size for subsequent transmissions, allowing the connection to continue without manual intervention.
PMTUD relies on specific firewall and forwarding configurations to function correctly across hybrid connections. VPC firewall rules must allow ICMP for IPv4 or ICMPv6 for IPv6 traffic from sources that match the original packet destinations; to simplify this, allowing ICMP and ICMPv6 from all sources is a common practice. Additionally, forwarding rules for internal passthrough Network Load Balancers and internal protocol forwarding must use the L3_DEFAULT protocol so they can process both the ICMP messages used for PMTUD and the protocol of the original packet being transmitted.
Fragmentation introduces performance overhead because routers must break larger packets into smaller segments, consuming CPU resources at each hop along the path. When PMTUD fails due to blocked ICMP messages, packets may be silently droppedâa condition known as blackholingâwhere the sender never learns that its packets are too large and cannot reach the destination. This makes proper firewall configuration critical for hybrid connections using Cloud Interconnect or HA VPN, where the network path crosses multiple administrative boundaries.
VPC networks support MTU values ranging from 1300 to 8896 bytes, with a default of 1460 bytes. For workloads using Cloud Interconnect or HA VPN, the MTU must account for the entire path between the on-premises network and Google Cloud, including any intermediate networks that may have lower MTU limits. When configuring MTU for hybrid connections, you should verify that all network segments along the path support the selected MTU value to avoid fragmentation or blackholing issues.
The Cloud VPN gateway has a fixed MTU size of 1460 bytes. This size accommodates the standard Ethernet MTU of 1500 bytes minus the 40 bytes of overhead added by the IPsec tunnel encapsulation (IP header, ESP header, and optional authentication trailer). For traffic to flow successfully from an on-premises network through a Cloud VPN tunnel to a Google Cloud Virtual Private Cloud (VPC), the peer VPN gateway on the on-premises side must be configured to support this same 1460-byte MTU. If the on-premises gateway or any intermediate network device uses a smaller MTU, packets exceeding that size will be fragmented or dropped, degrading performance. Therefore, validating and configuring consistent MTU settings across the entire path is a prerequisite for a functional hybrid connection.
In resilient architectures using HA VPN or Cloud Interconnect, MTU configuration must be validated not just for a single active path but for all potential failover paths. For HA VPN with active/active or active/passive tunnels, the same MTU (1460 bytes) applies to all tunnels in a gateway. When designing for failover, you must ensure the peer gateway's configuration for the backup tunnel also supports the correct MTU, so traffic continues without fragmentation after a failover event. A common pitfall occurs when multiple HA VPN gateways are configured in an active/passive mode; passive tunnels are not used for failover until all active tunnels on all gateways have failed, which can create unexpected bandwidth loss and complicate MTU validation across the broader topology.
For Cloud Interconnect (Dedicated or Partner), the MTU is typically 1500 bytes for the underlying physical connection, but the effective MTU for workloads must consider any additional encapsulation used in the network design, such as VLAN tags. The path through the interconnect and into the VPC must not introduce a lower MTU bottleneck. When using Cross-Site Interconnect for Layer 2 connectivity between on-premises sites over Google's network, the MTU is configurable and supports Jumbo frames (up to 9000 bytes) in VLAN mode, but this must be uniformly configured on all connected sites and matched to the capabilities of the attached on-premises routers.
Validation of MTU consistency is performed using diagnostic tools like ping with the Don't Fragment (DF) bit set and traceroute. You send ICMP echo requests with increasing packet sizes and the DF bit set from a source instance in Google Cloud to a destination in the on-premises network (or vice-versa). If a packet exceeds the MTU of any hop along the path, an ICMP "Fragmentation Needed" message should be returned, identifying the limiting MTU. Continuous validation should be performed across all redundant paths, including passive VPN tunnels or secondary Cloud Interconnect attachments, to ensure failover does not introduce MTU-related issues. This testing confirms the real-world data plane matches the designed configuration and helps avoid performance degradation during normal operation or failover scenarios.
The maximum transmission unit (MTU) defines the size, in bytes, of the largest packet that a network protocol layer can transmit, covering both packet headers and payload data. When network traffic traverses hybrid networking paths such as Cloud VPN or Cloud Interconnect, encapsulation protocols append additional header bytes to each packet. Calculating the effective MTU requires subtracting this encapsulation overhead from the underlying path MTU to prevent packet fragmentation, latency spikes, or dropped traffic. Properly aligning MTU configurations across virtual machine (VM) interfaces, Cloud Router instances, and external gateways ensures reliable and high-performance hybrid communication.
Encapsulation adds extra protocol headers to network packets, which reduces the total space remaining for user payload data. For workloads communicating across hybrid connections without encapsulation, systems standardly operate using a 1500-byte MTU setting across on-premises endpoints, internet paths, and private cloud interfaces. When traffic requires tunneling through VPN protocols such as IPsec or GRE, the effective MTU must be reduced based on specific endpoint configurations, generally resulting in an interface MTU between 1350 and 1390 bytes or lower. For environments utilizing VMware HCX, the default MTU setting on the HCX uplink profile is 1440 bytes.
Traffic type also dictates how MTU adjustments must be handled to protect application performance. While TCP-based traffic flows generally resolve fragmentation issues through Maximum Segment Size (MSS) clamping, large-packet UDP traffic cannot rely on MSS negotiation. For fragmentation-sensitive UDP traffic traversing public interfaces or internet paths, an MTU setting of 1370 bytes or lower prevents packet drops. Applying calculated MTU values directly to workload network interfaces prevents gateways from performing costly packet fragmentation along the hybrid network path.
A Virtual Private Cloud (VPC) network supports configurable MTU values ranging from the standard default of 1460 bytes up to a maximum allowed value of 8896 bytes. Increasing the VPC MTU to 8896 bytes enables jumbo frames, which reduces header overhead and increases payload data throughput by up to 10% compared to the 1460-byte default. Standard private clouds support internal endpoint MTU settings up to 8800 bytes, whereas stretched private clouds support settings up to 8600 bytes. Dedicated GPU instancesâsuch as A2, G2, and A4 seriesârecommend an 8896-byte MTU, while specialized machine types like A3 with header split enabled require an 8244-byte MTU.
To use jumbo frame MTU sizes exceeding standard limits on Google Cloud VMs, the guest operating system must deploy the Google Virtual NIC (gVNIC) driver version 1.3 or later. Operating systems without native jumbo frame support can install updated gVNIC drivers manually to access high-bandwidth networking tiers. Verifying the VPC network MTU ensures that VM interfaces do not attempt to transmit packet frames that exceed the underlying network's physical layer capabilities.
Consistent MTU configuration across VLAN attachment resources and hybrid gateways prevents asymmetrical packet handling and unexpected data drops. When establishing physical or partner connections through Cloud Interconnect, network architects must configure the exact same MTU setting across all active VLAN attachments. For Cloud VPN deployments, cryptographic negotiations directly influence packet overhead and the resulting effective MTU. Because cipher negotiation can change dynamically during Security Association (SA) rekeying, Cloud VPN tunnels and peer VPN gateways must be configured to propose and accept only one stable cipher algorithm per cipher role.
Fragmentation occurs when a router breaks a large packet into smaller segments to fit a smaller MTU, which consumes CPU resources at each hop. Blackholing occurs when Path MTU Discovery (PMTUD) fails because ICMP messages are blocked, causing the sender to never learn that its packets are too large, so the packets are silently dropped without any notification to the sender.
Start with the underlying path MTU (typically 1500 bytes for Ethernet) and subtract the IPsec tunnel encapsulation overhead, which is 40 bytes (IP header, ESP header, and optional authentication trailer). This results in an effective MTU of 1460 bytes for the Cloud VPN gateway. For other VPN protocols like IPsec or GRE, the overhead varies based on specific endpoint configurations, generally resulting in an interface MTU between 1350 and 1390 bytes.
Use jumbo frames (up to 8896 bytes) when workloads require high throughput and all network segments along the hybrid path support the larger MTU, including the on-premises routers, Cloud Interconnect attachments, and VPC network. Jumbo frames reduce header overhead and can increase payload data throughput by up to 10% compared to the 1460-byte default. However, you must ensure the guest operating system uses the Google Virtual NIC (gVNIC) driver version 1.3 or later to support jumbo frame sizes.
Professional Cloud Network Engineer
Prepare and test your skills
Prepare and test your skills
StudyCerts
StudyCerts