Professional Cloud Network Engineer

Configuring Self-Managed Network Virtual Appliance and Packet Mirroring

Routing and Inspecting Inter-VPC Traffic Using Multi-NIC Virtual Machines

A multi-NIC virtual machine is a single Compute Engine instance that connects to multiple VPC networks through separate network interfaces, allowing it to inspect and control traffic between different networks. Each network interface attaches to a different VPC or subnet, with one interface typically connected to an untrusted transit network and others connected to trusted internal networks. The appliance must have IP forwarding enabled on its network interfaces to forward traffic between them, and instance-level firewall rules must be configured to allow the intended traffic flows.

The hub-and-spoke topology is the most common design for multi-NIC appliance deployments, where a central routing VPC acts as the hub containing a load-balanced group of network virtual appliances (NVAs), while workload VPCs connect as spokes through VPC Network Peering. Traffic between spoke VPCs is forced through the NVAs for security inspection by using custom route advertisements and policy-based routing. To establish connectivity, you create VPC Network Peering connections between the routing VPC and each workload VPC, enabling Export custom routes on the routing VPC and Import custom routes on each spoke VPC.

High availability for NVA clusters is achieved using internal passthrough Network Load Balancers paired with managed instance groups (MIGs). The load balancer continuously monitors backend instance health using health checks, dynamically routing traffic away from failed or degraded appliances. When HA VPN gateways connect VPC networks directly, deploying both gateways in the same region provides a 99.99% availability service level agreement using a dual-tunnel setup, while deploying across different regions reduces the SLA to 99.9%.

Operational monitoring integrates Google Cloud Observability, VPC Flow Logs, and Flow Analyzer to track traffic volumes, packet statistics, and anomalous flows. VPC Flow Logs captures 5-tuple network flow samples—source IP, destination IP, source port, destination port, and protocol—from VM network interfaces. Flow Analyzer and Log Analytics query these logs to aggregate traffic by source and destination, helping verify that traffic flows through the inspection appliance as intended.

Configuring an Internal Load Balancer as a Next Hop for HA Multi-NIC VM Routing

An internal passthrough Network Load Balancer becomes a routing next hop when referenced by a custom static route. When you create a static route with an internal passthrough Network Load Balancer as the next hop, packets matching the route's destination are forwarded to the load balancer's frontend IP address, which then distributes these packets among its backend VMs using equal-cost multi-path (ECMP) distribution. The backend VMs receive packets with their original source and destination IP addresses, protocol, and ports unchanged—a behavior known as direct server return (DSR).

You can specify a next hop internal passthrough Network Load Balancer using three methods: by forwarding rule name and region, by forwarding rule resource link, or by the load balancer's internal IP address. The forwarding rule's VPC network must match the route's VPC network when specifying by name and region or by resource link. All backend VMs must have IP forwarding enabled (--can-ip-forward=True) on their network interfaces, and appropriate firewall rules must allow traffic from routed packet sources to routed packet destinations.

The global access setting on the internal passthrough Network Load Balancer determines whether the next hop is reachable from all regions or only from the same region. With global access disabled (the default), packets sent from another region to a route using an internal passthrough Network Load Balancer next hop are dropped. With global access enabled, the load balancer next hop is accessible from any region in the VPC network.

Health checks for multi-NIC virtual appliance backends monitor interface availability by periodically probing a designated port and protocol on each backend instance. A failover policy on the load balancer defines how traffic switches automatically between primary and backup backend instance groups—when the proportion of healthy primary instances falls below the configured failover ratio, traffic redirects to the failover backend pool. Session persistence uses a connection tracking table and consistent hashing to ensure packets belonging to an established connection consistently reach the same backend VM interface.

When multiple static routes have the same destination and priority but different internal passthrough Network Load Balancer next hops, Google Cloud does not distribute traffic among them using ECMP. Instead, Google Cloud selects a single next hop using a deterministic internal algorithm. To avoid this ambiguity, you should use unique network tags for each route or ensure routes have different priorities.

Configure Policy-Based Routes for HA Multi-NIC VM Routing

Policy-based routing in Google Cloud directs network traffic using custom packet characteristics rather than relying only on destination addresses. A policy-based route matches packets against configured criteria—such as source IP ranges, destination IP ranges, and IP protocol versions—and forwards them to a designated next hop like an internal passthrough Network Load Balancer. Google Cloud evaluates routes in a fixed order: special routing paths first, policy-based routes second, subnet routes third, and custom static or dynamic routes last.

Symmetric routing ensures that bidirectional network traffic passes through the exact same NVA instance in both directions to maintain stateful connection tracking. Google Cloud uses symmetric hashing on the internal passthrough Network Load Balancer, calculating consistent five-tuple packet header hashes so matching forward and return traffic reaches the same backend NVA without needing source network address translation (SNAT). This is essential for firewall and inspection appliances that need to see both directions of a connection.

To steer incoming traffic from Cloud Interconnect or HA VPN gateways, administrators define policy-based routes within the transit VPC that match the designated source and destination subnets. To prevent packets leaving an NVA from looping back into the appliance, administrators configure a higher-priority skip-policy-based route targeting the NVA interfaces with a next hop set to DEFAULT_ROUTING, which lets the inspected packet drop through to standard VPC routing.

Guest operating system routing configuration ensures that traffic entering an appliance on a specific interface exits back out through that same interface's default gateway. Because Google Cloud assigns a default gateway exclusively to nic0, the guest OS requires custom policy routing tables and source-based routing rules for all secondary interfaces. Inside the Linux kernel, administrators must enable IP forwarding by setting net.ipv4.ip_forward or net.ipv6.conf.all.forwarding to 1, and the VM instance configuration must enable the --can-ip-forward flag at creation time.

Developing a Strategy for Out-of-Band Network Security Integration

Out-of-band network security integration copies production traffic from a consumer VPC network to a producer VPC network for inspection by security appliances without impacting production workload performance. This approach uses VPC Packet Mirroring to create replica packets that are analyzed for threats while maintaining the original traffic flow. The architecture involves a consumer VPC (the source network whose traffic is being copied) and a producer VPC (the separate network that hosts the collector infrastructure).

VPC Packet Mirroring is a Google Cloud service that copies network packets from a source VPC network to a collector appliance for out-of-band security inspection. A Packet Mirroring policy has three key parts: a collector (the destination specified as the IP address of an internal passthrough Network Load Balancer), filters (to select traffic by protocol, IP address ranges, and direction), and mirrored resources (the source instances defined by subnets or network tags). The mirroring process creates a replica of matching packets, encapsulates them in GENEVE protocol using UDP port 6081, and sends them to the internal load balancer.

The internal passthrough Network Load Balancer acts as the central traffic collector and distributor for the virtual appliance instances. Its backend service must be configured with a protocol of UDP or UNSPECIFIED to accept raw network flows. When receiving GENEVE-encapsulated traffic from packet mirroring, the load balancer's forwarding rule must use the UDP protocol on destination port 6081. Each NVA's network interface must be in the same VPC network as the load balancer's backend service.

Network Security Integration (NSI) provides an out-of-band method to integrate third-party inspection appliances with Google Cloud networks using a producer-consumer model. The service producer network contains scalable third-party appliances deployed as backends to an internal passthrough Network Load Balancer, and the consumer network uses firewall policies to select specific traffic and redirect it to an endpoint group. This approach provides comprehensive visibility into workload traffic and supports advanced network security protections without requiring changes to routing policies.

Cloud IDS is a Google-managed intrusion detection service that provides threat detection by creating a Google-managed peered network with mirrored VM instances. It works by mirroring traffic from the consumer VPC to Google-managed IDS VMs, where advanced threat detection technologies analyze the traffic. Each IDS endpoint can inspect traffic from any zone in its region, with a maximum inspection capacity of 5 Gbps per endpoint and support for traffic spikes up to 17 Gbps.

Configuring Packet Mirroring for VPC Traffic Toward Self-Managed Collectors

Configuring Packet Mirroring involves creating policies that define which traffic to copy and where to send it for inspection. The collector infrastructure receives and processes the mirrored traffic through an internal passthrough Network Load Balancer deployed in a separate producer VPC network. This load balancer distributes the GENEVE-encapsulated mirrored packets across a backend group of self-managed network virtual appliances (NVAs), such as intrusion detection systems.

For performance and cost, the architecture uses zonal mirroring deployments where the load balancer's forwarding rule and the backend NVAs must be in the same zone as the source VM, otherwise the packet is not mirrored. Managed instance groups (MIGs) provide the elasticity and redundancy needed to scale collector appliance pools horizontally as traffic increases. Administrators must size the NVA instances with sufficient CPU, memory, and network bandwidth to handle peak mirrored traffic.

Mirroring rules within network firewall policies use the actions MIRROR, DO_NOT_MIRROR, and GOTO_NEXT to control which traffic is copied for security analysis. For ingress traffic, standard firewall rules are evaluated first, and only allowed traffic is then evaluated against mirroring rules. For egress traffic, mirroring rules are evaluated first. Security telemetry from Cloud IDS and packet mirroring deployments can be exported to centralized Security Information and Event Management (SIEM) platforms through Cloud Logging.

Key Facts

• A hub-and-spoke topology uses a central routing VPC with NVAs as the inspection point, forcing all inter-spoke traffic through the appliance cluster via custom route advertisements and policy-based routing.
• An internal passthrough Network Load Balancer acts as a next hop for custom static routes, distributing traffic across health-checked backend VMs using direct server return (DSR) which preserves original packet headers.
• Symmetric hashing on an internal passthrough Network Load Balancer calculates a consistent five-tuple hash so bidirectional traffic traverses the same multi-NIC network virtual appliance instance without source NAT.
• VPC Packet Mirroring clones network packets from a consumer VPC and sends them to an internal passthrough Network Load Balancer for out-of-band inspection by network virtual appliances (NVAs).
• Google Cloud evaluates routes in a fixed order: special routing paths first, policy-based routes second, subnet routes third, and custom static or dynamic routes last.

Frequently Asked Questions

What is the difference between policy-based routing and custom route advertisements in multi-NIC appliance deployments?

Policy-based routing overrides the default VPC routing by specifying a destination range and next-hop target (the internal load balancer), while custom route advertisements export routes from the routing VPC to spoke VPCs so they know to send traffic to the NVA cluster. Policy-based routes are created in the spoke VPCs themselves, whereas custom routes are imported by the spokes from the routing VPC.

What is the difference between the consumer VPC and the producer VPC in Packet Mirroring?

The consumer VPC is the source network whose traffic is being copied for inspection. The producer VPC is the separate network that hosts the collector infrastructure, including the internal passthrough Network Load Balancer and the backend network virtual appliances (NVAs) that analyze the mirrored traffic.

Why is symmetric hashing necessary on internal load balancers deployed with multi-NIC NVAs?

Symmetric hashing calculates identical five-tuple hash results for both directions of a flow. This ensures that forward and return packets are sent to the exact same backend appliance VM, preserving stateful connection tracking for firewalls and inspection appliances without requiring source network address translation.

How does Cloud IDS differ from a self-managed NVA solution using Packet Mirroring?

Cloud IDS is a fully Google-managed intrusion detection service where Google provisions and manages the inspection endpoints, simplifying operations. A self-managed NVA solution using Packet Mirroring gives you full control over the choice of security appliance, its configuration, and scaling, but requires you to manage the underlying Compute Engine instances and software.