Professional Cloud Network Engineer

Implement Symmetric Routing in Multi-NIC Appliance Deployments

Symmetric routing ensures that bidirectional network traffic passes through the exact same network virtual appliance (NVA) instance in both directions to maintain stateful connection tracking. In Google Cloud, this architecture connects multiple Virtual Private Cloud (VPC) networks using multi-NIC virtual machines (VMs), internal passthrough Network Load Balancers, and policy-based routes (PBRs). Combining Google Cloud routing controls with guest operating system (OS) source-based routing provides automated traffic distribution, high availability failover, and consistent traffic inspection across network perimeters.

Multi-NIC Architecture and VPC Segmentation

A multi-NIC architecture attaches separate virtual network interfaces on a single VM to isolated VPC networks that define distinct security zones. An appliance VM typically attaches its primary network interface (nic0) to an untrusted transit VPC network handling external hybrid connections, while secondary network interfaces (such as nic1) connect to trusted internal application VPC networks. An internal passthrough Network Load Balancer resides in each VPC network to distribute incoming traffic across a managed instance group of backend NVAs. Google Cloud uses symmetric hashing on the internal passthrough Network Load Balancer, calculating consistent five-tuple packet header hashes so matching forward and return traffic reaches the same backend NVA without needing source network address translation (SNAT).

Policy-Based Routing and Next-Hop Steering

Policy-based routes override standard routing decisions by matching specific packet characteristics—including source IP ranges, destination IP ranges, and IP protocols—before evaluating standard routes. Google Cloud evaluates policy-based routes strictly by numerical priority, choosing the highest-priority matching route to forward packets to an internal passthrough Network Load Balancer next hop. To steer incoming traffic from Cloud Interconnect or HA VPN gateways, administrators define policy-based routes within the transit VPC that match the designated source and destination subnets. To prevent packets leaving an NVA from looping back into the appliance, administrators configure a higher-priority skip-policy-based route targeting the NVA interfaces with a next hop set to DEFAULT_ROUTING, which lets the inspected packet drop through to standard VPC routing.

Guest Operating System Source-Based Routing Configuration

Guest operating system routing configuration ensures that traffic entering an appliance on a specific interface exits back out through that same interface's default gateway. Because Google Cloud assigns a default gateway exclusively to nic0, the guest OS requires custom policy routing tables and source-based routing rules for all secondary interfaces. Inside the Linux kernel, administrators must enable IP forwarding by setting net.ipv4.ip_forward or net.ipv6.conf.all.forwarding to 1. In Google Cloud, the VM instance configuration must also enable the --can-ip-forward flag at creation time, or the underlying virtualization platform drops packets whose source IP addresses do not match the assigned interface address.

Firewall Rules and Traffic Policy Enforcement

Firewall rules in multi-NIC appliance environments must permit end-to-end packet delivery in both ingress and egress directions. Ingress firewall rules protecting the NVA backends must allow the original client source IP ranges, because passthrough load balancers deliver packets without altering IP headers. Egress firewall rules associated with the NVA instances must permit routed packet destinations across all attached VPC subnets and external networks. Google Cloud health check probe IP ranges must also be allowed on all backend interfaces so the load balancer can detect unhealthy instances and redirect active traffic flows.

Integrate Internal Load Balancers as Next Hops for Resilient Failover

Using an internal passthrough Network Load Balancer as a next hop provides automated, resilient failover for traffic directed through a pool of third-party NVAs. This architecture allows a route to forward traffic destined for a specific network prefix to a health-checked pool of appliance VMs, maintaining high availability and stateful inspection.

Internal Passthrough Network Load Balancer as a Next Hop

An internal passthrough Network Load Balancer acts as a regional Layer 4 load balancer that routes traffic to backend VMs without altering packet headers. When configured as a route's next hop, the load balancer serves as the gateway for all matching traffic, forwarding packets directly to healthy backend appliance VMs. This approach establishes a single, stable next-hop IP address in the routing table, abstracting the individual backend VMs and simplifying failover operations.

Configuration for Resilient Failover

To establish resilient failover, the internal passthrough Network Load Balancer uses a backend service with a managed instance group containing appliance VMs distributed across multiple zones. A health check probes the backend VMs continuously, and the load balancer stops routing traffic to any VM that fails its health check. For multi-NIC deployments, configuring the load balancer to use symmetric hashing ensures that forward and return connections pass through the exact same backend VM for stateful inspection.

Policy-Based Routing Integration

This load balancing pattern is often used within a hub-and-spoke topology alongside policy-based routes to enforce strict traffic inspection paths. A central hub VPC hosts the load balancer and the appliance pool, while policy-based routes in spoke VPCs or on specific network interfaces steer traffic based on source and destination addresses directly to the load balancer IP. A skip-policy-based route applied to the appliance VM interfaces prevents inspected traffic from looping back to the load balancer, sending it instead to the VPC network's standard routing table.

Key Dependencies and Operational Order

Deploying this architecture requires a strict order of operations: the internal passthrough Network Load Balancer components (forwarding rule, backend service, health check, and instance groups) must be created before creating the route that references the load balancer as a next hop. The route and the load balancer forwarding rule must reside in the same VPC network unless connected through VPC Network Peering or Network Connectivity Center (NCC). A lifecycle dependency exists between these resources, meaning the load balancer cannot be deleted until all routes that reference it as a next hop are removed first.

Design Considerations and Trade-offs

Specific constraints govern internal load balancers used as next hops:

• The global access setting dictates routing reach: when disabled, only resources in the same region can use the next hop; when enabled, resources in all regions across the VPC network can access it.
• A static route's destination range cannot overlap with or be more specific than an existing subnet route.
• If all backend VMs fail their health checks, the route remains active and the load balancer continues to forward traffic across unhealthy backends according to its distribution algorithm rather than dropping the route.
• A forwarding rule configured with a shared internal IP address (--purpose=SHARED_LOADBALANCER_VIP) cannot be used as a next hop because the IP address must uniquely reference a single load balancer.

Configure Policy-Based Route Match Criteria and Routing Precedence

Policy-based routing in Google Cloud directs network traffic using custom packet characteristics rather than relying only on destination addresses. A policy-based route matches packets against configured criteria, such as source and destination IP ranges, and forwards them to a designated next hop like an internal passthrough Network Load Balancer.

Policy-Based Route Match Criteria

Policy-based routes evaluate packets using three primary criteria: source IP ranges, destination IP ranges, and IP protocol versions (IPv4 or IPv6). The source IP range specifies the origin address space that triggers the policy, while the destination IP range targets the intended endpoint. Evaluation occurs in order of route priority, where the highest-priority matching route forwards the packet and all lower-priority policy-based routes are ignored; if no match is found, evaluation continues down the sorted priority list.

Route Priority and Ordering Against VPC Routing Tables

Google Cloud applies routing logic in a defined evaluation sequence: special routing paths, policy-based routes, subnet routes, and finally custom static or dynamic routes. Because policy-based routes are evaluated before standard subnet routes and custom static routes, they can intercept specific flows before standard routing applies. When multiple policy-based routes share the exact same priority, Google Cloud selects one using an internal algorithm, making unique route priorities essential to avoid ambiguous routing. Setting the skip option on a policy-based route instructs Google Cloud to stop evaluating other policy-based routes and proceed straight to subnet route evaluation.

Interaction with Subnet Routes and Peering Routes

Policy-based routes interact hierarchically with the underlying VPC routing tables. When a packet matches an active policy-based route, Google Cloud forwards the packet to the specified next hop and ignores all non-policy-based routes. If no policy-based routes match, Google Cloud evaluates subnet routes (including local, peering, and NCC subnet routes); if a packet matches a local subnet route, that subnet route is used exclusively, meaning policy-based routes cannot override standard subnet routes for local destinations within the same VPC network.

Policy-Based Routes for HA Multi-NIC VM Routing

In hub-and-spoke topologies with multi-NIC NVAs, policy-based routes steer traffic from spoke VPCs through the appliances located in the central hub VPC. Applying untagged policy-based routes with broad source and destination CIDR ranges in the transit VPC captures bidirectional traffic, while regional policy-based routes direct incoming traffic from Cloud Interconnect to the internal load balancer. Once the NVAs process the packets, skip-policy-based routes applied to the appliance interfaces send the traffic to standard VPC routing, and static routes in the application VPCs direct return traffic back to the NVA load balancers to complete the symmetrical path.

Key Facts

• Symmetric hashing on an internal passthrough Network Load Balancer calculates a consistent five-tuple hash so bidirectional traffic traverses the same multi-NIC network virtual appliance instance without source NAT.
• Google Cloud evaluates routes in a fixed order: special routing paths first, policy-based routes second, subnet routes third, and custom static or dynamic routes last.
• A skip-policy-based route configured with a next hop of DEFAULT_ROUTING allows inspected packets leaving an appliance interface to bypass policy-based routing and follow standard VPC routes.
• An internal passthrough Network Load Balancer cannot be deleted while it is still referenced as a next hop by any static or policy-based route.
• Virtual machine instances acting as appliances must have the --can-ip-forward flag enabled in Google Cloud and IP forwarding enabled in the guest operating system kernel to route packets.

Frequently Asked Questions

Why is symmetric hashing necessary on internal load balancers deployed with multi-NIC NVAs?

Symmetric hashing calculates identical five-tuple hash results for both directions of a flow. This ensures that forward and return packets are sent to the exact same backend appliance VM, preserving stateful connection tracking for firewalls and inspection appliances without requiring source network address translation.

What happens if all backend VMs behind an internal load balancer next hop fail their health checks?

If all backend VMs become unhealthy, the route referencing the load balancer remains active in the VPC routing table. The load balancer continues forwarding traffic across the unhealthy backends using its standard distribution algorithm rather than withdrawing or dropping the route.

How do policy-based routes interact with local VPC subnet routes?

Policy-based routes are evaluated before subnet routes, but they cannot override local subnet routing for destinations inside the same VPC network. If a packet matches a policy-based route, it goes to the specified next hop; if it does not match, standard subnet routes handle local, peering, and Network Connectivity Center destinations exclusively.