Connectivity Tests is a diagnostics tool within the Network Intelligence Center that analyzes network configuration and, for supported paths, performs live data plane analysis to check connectivity between endpoints like VMs, GKE clusters, or Cloud SQL instances. It helps you determine whether a connectivity failure is caused by a routing problem (the packet cannot find a path to the destination) or a firewall problem (the packet is blocked by a security rule).
Configuration analysis simulates a packet’s expected forwarding path through your Google Cloud Virtual Private Cloud (VPC) network by modeling the network as a finite state machine. The analysis checks each logical step a packet would take, starting from the source endpoint. It first evaluates egress firewall rules to see if the source is allowed to send the packet. Then, it attempts to match a route for the destination IP address. If multiple valid routes exist (for example, several custom routes or a mix of policy-based and regular routes), the test returns all possible paths because the actual data plane selection method internal to Google Cloud is not deterministic from the configuration alone. If the packet passes the egress check and a valid route is found, the analysis proceeds to check the next hop, which could be an internal load balancer, a VPN tunnel, or a peered VPC network. The packet continues through this simulated path, checking relevant ingress firewall rules at the destination. A test result of Reachable means the configuration analysis found a complete, allowable path from source to destination. A result of Unreachable indicates the packet was dropped at a specific step, which the detailed results will show.
A routing issue is indicated when the analysis shows the packet is dropped because no valid route exists to forward it toward the destination. In the detailed trace results, you will see a step where the packet’s state is Dropped due to a route mismatch. This can happen if a necessary custom route is missing, a route’s next hop is incorrect or down (like a Cloud VPN tunnel in a down state), or if dynamic routes from a Cloud Router are not being advertised or learned properly. For tests involving connectivity to on-premises networks or other clouds via Network Connectivity Center, the tool can only analyze the configuration within Google Cloud; it assumes routes from the external network are valid, so a routing problem would be isolated to the Google Cloud side of the path.
A firewall issue is indicated when the analysis shows the packet is explicitly blocked by a VPC firewall rule, a hierarchical firewall policy, or a global network firewall policy. The detailed results will specify at which hop the firewall rule denied the packet, and will identify the specific rule. An egress firewall rule blocks traffic leaving the source VM’s network interface. An ingress firewall rule blocks traffic entering the destination network or VM. The tool simulates firewall connection tracking for return traffic: if the test simulates an outbound connection, subsequent return traffic that would normally be denied by an ingress rule is allowed in the simulation, because the connection was initiated from inside the VPC network.
For supported paths, Connectivity Tests also performs live data plane analysis, sending real packets between the hypervisors of the source and destination VMs. This test validates the actual data plane and provides baseline latency and packet loss metrics. A mismatch between the configuration analysis result (e.g., Reachable) and the live data plane result (e.g., packets are dropped) can highlight issues not captured in the static configuration, such as data plane bugs or resource health problems. However, if the configuration analysis itself shows an Unreachable result due to a firewall or route, the live test may not run, as the configuration must theoretically allow the traffic first.
When testing connectivity to Google-managed services like Cloud SQL or GKE control planes, Connectivity Tests can analyze the path up to the Google-owned VPC network but cannot provide configuration details within that network. If the test fails with a generic error like DROPPED_INSIDE_CLOUD_SQL_SERVICE, it indicates a potential configuration issue within the Google-managed service project. To differentiate this from a user-side routing or firewall issue, examine the test trace: if the packet successfully passes your egress firewall, matches a route to the peered service network, and passes the Google network edge, then the failure is inside the service. If the packet is dropped earlier in the trace within your own VPC, the issue is in your routing or firewall configuration.
Connectivity Tests is a diagnostics tool within the Network Intelligence Center that lets you check connectivity between network endpoints by analyzing your configuration and, in some cases, performing live data plane analysis. An endpoint is a source or destination of network traffic such as a VM instance, Google Kubernetes Engine (GKE) cluster, load balancer forwarding rule, or an IP address on the internet. The tool helps troubleshoot network connectivity issues caused by unintended inconsistent configurations, obsolete configurations from network changes or migrations, and configuration errors for various network services.
When analyzing network configurations, Connectivity Tests simulates the expected forwarding path of a packet through your VPC network, Cloud VPN tunnels, or VLAN attachments using an abstract state machine model. The analysis can take many possible paths depending on the VPC network services and features configured, such as Cloud VPN tunnels, VPC networks, Google Cloud load balancers, or peered VPC networks. Because a test packet can traverse many possible paths through the configuration, the analysis follows one of many finite states until the packet is delivered or dropped. When Connectivity Tests matches several routes according to route precedence, it returns all possible routes but cannot determine which specific route Google Cloud will choose, as that method is internal and subject to change.
For some connectivity scenarios, Connectivity Tests also performs live data plane analysis, which sends packets over the actual data plane to validate connectivity and provides baseline diagnostics of latency and packet loss. If the route supports this feature, each test includes a live data plane analysis result. However, live data plane analysis performs one-time verification for diagnostic purposes and is not intended for continuous monitoring. For continuous monitoring of connectivity and packet loss, the Performance Dashboard should be used instead. Live data plane analysis supports specific configurations including VPC Network Peering, Shared VPC, alias IP ranges, external and internal IP addresses, custom routes, load balancers with specific backends, and ingress firewall rules.
To construct a test scenario, you specify source and destination endpoints using a 5-tuple without the source port. For source endpoints, you can specify a VM instance with its IP address, an IP address alone (marking whether it’s a Google Cloud address), or a GKE cluster. For destination endpoints, you can specify a VM instance, an IP address, a load balancer, a Google-managed service endpoint, or a Private Service Connect endpoint. When testing to or from a VM with multiple network interfaces, you must specify which network interface to use by providing the IP address and network URI, the IP address and VM URI, or the network URI and VM URI.
Connectivity Tests supports testing connectivity for the following traffic flows: VM instances to and from the internet, VM instance to VM instance, from Google Cloud to and from on-premises networks, between two on-premises networks connected through Network Connectivity Center, and between two Network Connectivity Center VPC spokes. You can also test connectivity between a VM instance and a GKE control plane endpoint, between a VM instance and a Google network edge location, and between a VM and a load balancer. The supported protocols are TCP and UDP. The configuration analysis supports VPC networking features including VPC networks, VPC Network Peering, Shared VPC, Private Google Access, alias IP ranges, custom routes imported from peered VPC networks, VPC firewall rules, regional and hierarchical firewall policies, policy-based routes, Private Service Connect, and instances with IPv6 addresses.
When testing Google-managed services such as Cloud SQL and GKE, Connectivity Tests allocates resources in projects and VPC networks that Google owns and manages, where customers don’t have permission to access these resources. The configuration analysis can still run a test and provide an overall reachability result, but it doesn’t provide details for the tested resources in the Google-owned project. By default, Connectivity Tests attempts to run a test using the private IP address of the Google-managed service endpoint and the VPC Network Peering connection between your network and the Google-owned network. If the endpoint doesn’t have a private IP address, Connectivity Tests uses the public IP address. When testing through a public IP address, Connectivity Tests doesn’t analyze the configuration of the resources inside the Google-owned project.
Connectivity Tests analyzes the VPC network path configuration between two endpoints and shows whether the programmed configuration should allow traffic or not. If traffic is not allowed, the results show whether a Google Cloud egress or ingress firewall rule is blocking the traffic or if a route isn’t available. The test examines the configuration of the VPC network only and does not test the operating system firewall, operating system routes, or server software on the VM. When the results indicate the connection is dropped by a firewall rule, you must determine if your intended security setup should allow the connection. If the traffic should be allowed but isn’t, check the always blocked traffic list and review your firewall policies to create or modify rules to allow the connection.
Connectivity Tests has no knowledge of networks outside of Google Cloud, defined as on-premises networks in your data center, other cloud providers where you run resources, or hosts on the internet sending traffic to your VPC network. The configuration analysis doesn’t perform firewall connection tracking because the firewall connection table is located in the data plane for a VM instance and is inaccessible. The test also cannot test VM instances configured to act as routers, firewalls, NAT gateways, or VPNs in the data plane. Getting Connectivity Tests results can take from 30 seconds to up to 10 minutes depending on the size of your VPC network configuration and the number of Google Cloud resources used. Additionally, a status of “Packet could be delivered” does not guarantee that traffic can pass through the data plane, as the purpose is to validate configuration issues that can cause traffic to drop.
Remediating configuration errors based on Connectivity Tests analysis involves translating diagnostic trace results into corrective actions across firewall rules, routing tables, and hybrid connectivity resources. Connectivity Tests uses an abstract state machine to model packet processing in a VPC network and can validate paths through live data plane analysis. When the tool identifies dropped packets, administrators must modify specific Google Cloud configurations to align the programmed state with the intended network architecture.
Firewall misconfigurations occur when ingress or egress security policies unintentionally drop traffic along a simulated or live trace path. Connectivity Tests pinpoints the specific security rule responsible for dropping packets, allowing administrators to target and update the exact rule. To remediate firewall blocks, administrators modify existing VPC firewall rules, create new allow rules, or adjust hierarchical firewall policies and regional network firewall policies. Specific services require deliberate firewall permissions to establish necessary control plane and data plane communication. For example, when Router Appliance VM instances fail to establish Border Gateway Protocol (BGP) sessions with Cloud Router, administrators must ensure an ingress firewall rule explicitly permits TCP traffic on port 179. Similarly, hybrid deployments using regional external or internal Application Load Balancers require firewall rules that permit ingress traffic from regional proxy-only subnets or Google health check IP ranges.
Route misconfigurations occur when a packet cannot find a valid next hop or when routing tables do not propagate destinations across regions and hybrid connections. When Connectivity Tests reveals that packets are dropped due to missing routes between regions, administrators must switch the VPC dynamic routing mode from regional to global. For custom route priority issues, administrators must inspect the VPC routing table, update custom static routes, or adjust next-hop pointers—such as redirecting the default 0.0.0.0/0 route to an internal load balancer forwarding rule in transit hub topologies. In multi-network deployments using Network Connectivity Center (NCC), routing failures often stem from spoke filter settings or BGP configuration errors. If hub subnets are not advertised to on-premises networks, administrators must verify BGP route policies and update the include-export or exclude-export IP ranges on the VPC spokes. For Router Appliance BGP peering failures, administrators must correct IP address allocations by ensuring instances use internal RFC 1918 addresses within the same subnet as the Cloud Router, rather than link-local 169.254.x.x addresses.
Network address translation and load balancer misconfigurations arise when translation rules, backend attachments, or port allocations conflict with network reachability. When Connectivity Tests identifies drops in environments combining Private NAT and Private Service Connect (PSC) propagated connections, administrators must refine the NAT configuration. Configuring Private NAT with the --nat-all-subnet-ip-ranges flag causes traffic destined for propagated PSC connections to drop; administrators remediate this by specifying only overlapping subnets using the --nat-custom-subnet-ip-ranges flag. Workloads using Direct VPC egress or regional internet network endpoint groups (NEGs) require precise Cloud NAT gateway adjustments. When Cloud NAT is used with Cloud Run Direct VPC egress, administrators must set the endpoint type to ENDPOINT_TYPE_VM and size the minimum ports per instance to double the required count for Public NAT or quadruple it for Private NAT. After making any routing, firewall, or NAT remediation changes, administrators should wait between 20 and 120 seconds for Connectivity Tests to receive and incorporate configuration updates before rerunning the test.
Check the detailed trace results from Connectivity Tests. If the packet is dropped at a step where no valid route exists for the destination IP, it is a routing issue. If the packet is dropped at a step where a specific firewall rule is identified as blocking the packet, it is a firewall issue. The tool will show the exact rule name or the missing route.
Configuration analysis simulates the packet path using the programmed network configuration (firewall rules, routes, etc.) and returns a reachable or unreachable result. Live data plane analysis sends real packets between the hypervisors of the source and destination VMs to validate actual connectivity, providing baseline latency and packet loss metrics. Live analysis only runs when configuration analysis shows a reachable path.
You can test VM-to-VM, VM-to-internet, VM-to-on-premises, on-premises-to-on-premises through Network Connectivity Center, VM-to-GKE control plane, VM-to-Google network edge, and VM-to-load balancer flows. The supported protocols are TCP and UDP. The tool supports many VPC features including VPC Network Peering, Shared VPC, Private Google Access, alias IP ranges, custom routes, and firewall policies.
First, check the always blocked traffic list to see if the traffic is inherently denied. Then review your VPC firewall rules, hierarchical firewall policies, and regional network firewall policies. Create or modify allow rules to permit the specific source, destination, protocol, and port. For example, if a Router Appliance BGP session is failing, add an ingress firewall rule allowing TCP port 179 from the Cloud Router’s IP range.
Professional Cloud Network Engineer
Prepare and test your skills
Prepare and test your skills
StudyCerts
StudyCerts