Managing the Maximum Transmission Unit (MTU) in networks with multiple VPCs involves ensuring packets are sized correctly across all connected networks to prevent them from being dropped. The Path MTU Discovery (PMTUD) process automatically finds the largest packet size a network path can handle, but it needs specific conditions to work.
The MTU of a VPC network must work with every connected service. For traffic between instances in the same VPC or peered VPCs, the path's effective MTU is the smallest MTU of any VPC in the path. A key rule is that traffic sent from a VPC with a higher MTU to a peered VPC with a lower MTU will have its packets cut down to the lower size, which can cause problems for non-TCP protocols. For hybrid connections using Cloud Interconnect VLAN attachments, it is best to use the same MTU for all VLAN attachments on a single VPC and to match the VPC's MTU to this value. Using different MTUs, or using a VLAN attachment MTU that is smaller than the VPC's MTU, can lead to dropped packets, especially for non-TCP traffic.
Path MTU Discovery (PMTUD) lets hosts discover the maximum packet size for a path. When a router cannot forward a packet because it is too large and the "Don't Fragment" (DF) bit is set, it must send an ICMP "Fragmentation Needed" message (Type 3, Code 4 for IPv4) back to the source. For PMTUD to work in Google Cloud, all intermediate firewallsâincluding VPC firewall rules and firewall policiesâmust be configured to allow these specific ICMP messages. The source of these ICMP messages is the original packet's destination IP, so firewall rules must permit ICMP traffic coming from those destination addresses. Additionally, for internal passthrough Network Load Balancers to support PMTUD, their forwarding rules must use the L3_DEFAULT protocol so they can handle both the original traffic and the necessary ICMP replies.
Packet loss from MTU problems often happens when PMTUD fails because ICMP messages are blocked. To diagnose the issue, you must verify that ICMP "Fragmentation Needed" messages can travel from the point where the packet is too large all the way back to the original sender. This means checking all firewall rules and policies along that return path. For TCP traffic, Google Cloud provides a safety feature called MSS clamping on connection setup packets, which automatically adjusts the maximum segment size to fit the path's MTU. However, for UDP and other non-TCP protocols, there is no automatic adjustment, making a working PMTUD essential. The fix is to ensure firewall rules explicitly allow ICMP Type 3, Code 4 (for IPv4) or the equivalent ICMPv6 "Packet Too Big" messages from the relevant destinations.
The Maximum Transmission Unit (MTU) is the largest size, in bytes, of an IP packet that can be sent over a Virtual Private Cloud (VPC) network. Google Cloud VPC networks support MTU values from 1300 to 8896 bytes, with a default of 1460 bytes. Setting the VPC MTU to jumbo frame sizes, like the maximum 8896 bytes, reduces packet header overhead and can improve data throughput by up to 10% for high-performance workloads like Google Cloud Managed Lustre. You can check and update the VPC MTU using the Google Cloud CLI as your workload needs change.
Specialized Compute Engine machine types built for accelerators like GPUs need specific MTU values to maximize network performance. Machine series such as A4X, A4, A3 Ultra, A2, G2, G4, and GPU-enabled N1 instances perform best with an MTU of 8896 bytes. However, A3 Mega, A3 High, and A3 Edge machine types require a different MTU of 8244 bytes for their GPU network interfaces when header split is enabled. This specific 8244-byte size ensures incoming data frames fit efficiently across two standard 4 KB memory buffers in the operating system.
Google Cloud VMware Engine has different MTU rules based on how traffic is encapsulated and where it travels. Virtual machines communicating only within a standard private cloud can use MTUs up to 8800 bytes, while stretched private clouds support up to 8600 bytes. Traffic routed outside the private cloud without any encapsulation uses the standard 1500-byte MTU. For internet-bound UDP traffic that is sensitive to fragmentation, an MTU of 1370 bytes or lower is needed. Traffic that is encapsulated, such as over VPN tunnels or VMware HCX uplinks, generally requires an MTU between 1350 and 1390 bytes to account for the extra encapsulation headers.
The network interface inside a virtual machine's guest operating system (OS) must have its MTU setting match the underlying VPC network's MTU. If they do not match, the OS may discard packets, leading to poor performance. For instances using jumbo frames, the Google Virtual NIC (gVNIC) driver version 1.3.0 or later is required, while bare metal instances use the Intel Data Plane Function (IDPF) driver. Beyond matching the MTU, you can optimize throughput by enabling features in the guest OS like TCP segmentation offload (TSO) and large receive offload (LRO). Tools like ethtool can be used inside the VM to diagnose packet drops caused by buffer issues or MTU misconfigurations.
Choosing the correct Maximum Transmission Unit (MTU) size is the process of selecting the largest packet that can travel a network path without being broken into smaller pieces (fragmented). In Google Cloud, getting the MTU right is key for good performance, especially when connecting to on-premises networks via Cloud Interconnect or Cloud VPN.
When you set up a VPC network, you configure its MTU, which defines the largest IP packet it can transmit. Using the maximum value of 8896 bytes can improve network performance by up to 10% compared to the default 1460 bytes, because larger packets carry more data with less relative header overhead. You can set the MTU when creating a VPC and update it later, though changing an existing network's MTU requires careful planning. High MTU values are most beneficial for data-heavy workloads like high-performance computing or storage transfers.
Cloud Interconnect provides a dedicated physical link between your on-premises network and Google Cloud. These connections support jumbo frames with MTUs up to 8896 bytes, which greatly reduces overhead when moving large datasets. For consistent performance, it is recommended to use the same MTU value for all VLAN attachments connected to the same Interconnect. The capacity of each VLAN attachment can be configured, with a default of 10 Gbps, and the actual bandwidth depends on the capacity of the physical Interconnect connection.
Cloud VPN creates encrypted IPsec tunnels over the public internet to connect on-premises networks to Google Cloud. This encryption adds extra headers to each packet, reducing the amount of space left for actual data (the effective MTU). For virtual machines communicating through a Cloud VPN tunnel, the recommended MTU setting is between 1350 and 1390 bytes or lower. This lower value accounts for the 50-100 bytes of IPsec overhead. Using an MTU that is too large for VPN traffic causes packets to fragment, which hurts performance and increases CPU load on both ends of the connection.
TCP Maximum Segment Size (MSS) clamping is a technique to prevent fragmentation by automatically reducing the maximum data segment size that TCP endpoints agree to use. When enabled on a network device, it modifies the TCP header during connection setup to advertise a smaller segment size, ensuring packets never exceed the path's MTU. MSS clamping is very effective for fixing fragmentation issues with TCP traffic. For UDP traffic that is sensitive to fragmentation, such as large UDP packets sent over the internet, MSS clamping does not apply; instead, you should use a lower MTU of 1370 bytes or lower from the start.
The ideal MTU depends entirely on the path the traffic takes. For traffic between VMs within a single Google Cloud VPC (with no hybrid connectivity), the standard 1500-byte MTU is typical and works well. For traffic crossing a Cloud Interconnect to or from on-premises, jumbo frames up to 8896 bytes maximize throughput if all network devices in the path support them. For traffic going through a Cloud VPN IPsec tunnel, the MTU must be lowered to between 1350 and 1390 bytes to accommodate encapsulation headers. For large UDP packets sent to the internet, an MTU of 1370 bytes or lower helps avoid performance-degrading fragmentation.
The main risk is that packets sent from the VPC with the higher MTU will be truncated (cut down) to the lower MTU size when they enter the peered VPC. This truncation can cause packet loss or corruption for protocols other than TCP, which does not have a built-in recovery mechanism for this issue.
PMTUD most commonly fails because intermediate firewalls, including VPC firewall rules, are blocking the specific ICMP "Fragmentation Needed" messages that routers need to send back to the source host. These messages must be allowed for PMTUD to function and automatically determine the correct packet size for a path.
You should use jumbo frames for workloads that benefit from high throughput and transfer large amounts of data, such as high-performance computing (HPC), storage traffic (like Google Cloud Managed Lustre), or data transfers over Cloud Interconnect. They are also required for optimal performance on specific accelerator-optimized machine types like the A4X or G2 series.
Cloud Interconnect supports jumbo frames (up to 8896 bytes) for high-throughput, low-overhead data transfer. Cloud VPN, due to the added headers from IPsec encryption, requires a much lower MTU (1350-1390 bytes) to prevent fragmentation. Choosing the wrong MTU for either service can lead to significant performance degradation.
Professional Cloud Network Engineer
Prepare and test your skills
Prepare and test your skills
StudyCerts
StudyCerts