Network performance issues in Google Cloud originate from two fundamentally different scopes: problems affecting Google's global backbone infrastructure that impact multiple projects, or issues isolated to a specific project's configuration and resources. Understanding this distinction is critical for effective troubleshooting, as the resolution paths diverge significantly depending on the source.
The Performance Dashboard within Network Intelligence Center provides project-level visibility into network performance metrics. When troubleshooting packet loss or latency, the dashboard lets you examine specific zone pairs within your project, focusing on the path between source and destination resources you own. A project-scoped issue typically manifests as degraded performance between particular VM instances, within a specific region, or involving particular VPC configurations. For example, if your source or destination VM is being throttled while exceeding allowed egress or ingress bandwidth, the Performance Dashboard will show elevated latency or packet loss that correlates with the affected VM's traffic volume. You diagnose project-scoped issues by reviewing the Network Bytes metric on the VM instance details page and comparing it with the bandwidth limits described for the machine type.
Google-wide issues affect the global infrastructure that all Google Cloud customers share, including the backbone network, Google Front Ends (GFEs), and edge points of presence. When a widespread problem is affecting the network data plane, you should consult both your project's Performance Dashboard and the Google Cloud Status Dashboard. A Google-wide issue typically manifests as degraded performance across multiple zone pairs and potentially affects traffic between different projects or regions. The key indicator is when configuration analysis shows "Packet could be delivered" but live data plane analysis indicates partial packet loss across what should be healthy paths. These issues are outside your project's configuration and require Google to resolve them rather than internal troubleshooting.
When Connectivity Tests show a mismatch between configuration analysis and live data plane results, the scope determination follows a specific logic. First, verify whether recent VPC network configuration changes might have caused inconsistencyâif so, the issue is project-scoped and resolves by avoiding configuration changes during testing. Second, check for sporadic network programming issues by stopping and starting the affected VM, which addresses project-specific state problems. Third, analyze whether the source or destination VM is being throttled, which indicates a project-scoped resource constraint. Finally, if these project-level checks reveal nothing abnormal and the issue spans multiple zones or regions, consult the Google Cloud Status Dashboard to determine whether a Google-wide incident is affecting your traffic.
End-to-end latency consists of multiple segments, each falling under different scopes. Latency from the user to Google's network edge depends on the user's location and whether you use Premium or Standard Network Service Tierâthis segment is largely outside project control but represents Google-wide infrastructure. The latency between the edge point of presence and the Compute Engine region where your resources reside sits within Google's global network. Application latency, which includes processing time within your VM, is entirely project-scoped. When diagnosing latency issues, you must determine whether delays originate from the Google-wide portion of the path or from project-specific factors like load balancer configuration, VM processing, or VPC routing.
Interpreting Performance Dashboard metrics involves analyzing packet loss and latency data across Google Cloud to diagnose network issues. The Performance Dashboard provides both Google-wide and project-specific views, enabling you to determine if a performance problem is isolated to your project or part of a broader Google Cloud infrastructure event. You interpret the visualizationsâsuch as time-series graphs, percentile distributions, and anomaly indicatorsâto distinguish normal performance variations from critical degradation.
Packet loss metrics in the Performance Dashboard measure the percentage of packets failing to reach their destination between VM instances. The dashboard shows an average packet loss for up to 50 zone pairs across all of Google Cloud, providing a Google-wide baseline. For your project, packet loss is measured using active probes: UDP packets test internal IP addresses, and TCP packets test external IP addresses. The system's confidence in these measurements depends on your deployment scale; higher confidence requires more VMs per zone. The dashboard flags low-confidence data with an asterisk (*) or displays N/A when there is insufficient probe data. A key anomaly to note is that packet loss alone does not constitute downtime per the Google Cloud SLA; you must correlate it with application symptoms.
Latency metrics measure the round-trip time (RTT) for traffic between VM instances, between peered VPCs, and to internet endpoints. The dashboard displays the median latency for up to 50 zone pairs. Latency is calculated passively from sampled TCP traffic headersâthe same source as VPC Flow Logsâby measuring the time between a TCP sequence number (SEQ) and its corresponding acknowledgment (ACK). You must be aware of specific anomalies: in low-traffic environments, sixty-second probes can cause falsely high RTT readings if the sampling misses the ACK and counts a later data response. Additionally, project-specific latency data might not always align with the global Google Cloud latency data due to sampling differences. When analyzing, you should check if high RTT levels correspond with actual application-level delays.
The Performance Dashboard provides two distinct scopes for analysis: a Google-wide view and a project-scoped view. The Google-wide view shows performance for any zone pair across all of Google Cloud, regardless of whether your project has resources there, helping you establish a baseline. The project-scoped view shows metrics only for traffic involving your project's VMs, including those within a single VPC, between peered VPCs in the same project, and to internet endpoints. A key dependency exists for Shared VPC service projects: the dashboard only shows data for zones within that specific service project, not for traffic routed through the host project to another service project. You switch between these views using the Change Scope button or the project picker, which allows you to leverage an existing metrics scope for monitoring multiple projects.
The Performance Dashboard helps detect anomalies by visualizing deviations in time-series data for packet loss and latency. You are advised to create alerting policies based on predefined conditions; for example, an alert triggers when packet loss exceeds 5% for 5 minutes for any region pair. These alerts, managed through Cloud Monitoring, provide timely awareness of problems. When investigating anomalies, you must correlate dashboard metrics with other diagnostic data. For instance, a latency spike shown globally might indicate a broader Google Cloud issue, while a spike only in your project view points to an application or configuration problem within your scope. The recommended troubleshooting flow is to first check the Google-wide dashboard to rule out infrastructure issues, then drill into the project-specific view to isolate local causes.
Correlating performance findings with underlying network topology involves mapping diagnostic metrics from Performance Dashboard directly onto the structural elements visualized within Network Topology. By comparing broad packet loss and round-trip time (RTT) measurements against specific Virtual Private Cloud (VPC) paths, hybrid connections, and Google Kubernetes Engine (GKE) deployments, administrators can isolate whether performance degradations originate from the global Google network, regional infrastructure, or project configurations.
Performance Dashboard provides visibility into network latency and packet loss across the entire Google Cloud infrastructure as well as within specific project scopes. Users can evaluate Google-wide performance for up to 50 zone pairs across a selected time window or inspect project-scoped traffic between virtual machine (VM) instances and internet endpoints. Packet loss is measured actively using User Datagram Protocol (UDP) packets for internal IP addresses and Transmission Control Protocol (TCP) packets for external IP addresses. In contrast, latency metrics are gathered passively from actual TCP traffic samples, calculating the median elapsed time between a sent sequence number and the corresponding acknowledgment packet.
Network Topology provides an interactive visual map of network infrastructure, detailing relationships and real-time operational metrics across VPC networks, hybrid connectivity, and GKE deployments. When Performance Dashboard highlights elevated latency or packet loss for a specific region pair, administrators use Network Topology to examine the exact communication paths, throughput, and query-per-second metrics between connected workloads. The topology view allows filtering down to individual GKE clusters or peered entities, displaying latency distributions across the 50th, 95th, and 99th percentiles over time ranges spanning up to six weeks. This combined workflow reveals whether abnormal metrics stem from specific VM instances, pod communication bottlenecks, or intermediary network hops.
Project-specific latency analysis depends on traffic flowing within a single VPC network, across peered VPC networks within the same project, or toward external internet endpoints. In a Shared VPC architecture, Performance Dashboard displays metrics only for VM instances residing within the specific service project, meaning cross-service project traffic routed through the host project is not visible to either project. In environments with low traffic rates, latency sampling can occasionally misinterpret delayed application-layer responses as slow network round trips, artificially inflating RTT values. Additionally, metric confidence scales with probing density; zones with fewer deployed VM instances or under 400 probes per minute are flagged with low-confidence indicators or marked unavailable.
Performance Dashboard automatically exports time-series performance data to Cloud Monitoring for continuous tracking and custom alerting. Accessing these metrics requires the monitoring.timeSeries.list permission, which is granted through roles such as Monitoring Viewer (roles/monitoring.viewer), Monitoring Editor (roles/monitoring.editor), or Monitoring Admin (roles/monitoring.admin). Automated alerting policies can be configured to notify teams whenever packet loss exceeds a threshold of 5% for 5 minutes across any region pair. During severe zonal or regional outages, data streams for the affected zones temporarily drop out of both Network Topology and Performance Dashboard charts until service restoration occurs.
First, check the Google-wide Performance Dashboard view to see if the issue affects multiple zone pairs across different projects. If it does, consult the Google Cloud Status Dashboard for a confirmed incident. If the issue appears only in your project-scoped view, check for recent VPC configuration changes, VM throttling due to bandwidth limits, or sporadic network programming issues that can be resolved by stopping and starting the affected VM.
The dashboard flags low-confidence data with an asterisk (*) or displays N/A when there is insufficient probe data. This occurs when zones have fewer deployed VM instances or under 400 probes per minute. Higher confidence requires more VMs per zone, so you should add more instances to improve measurement accuracy.
When the Performance Dashboard shows elevated latency or packet loss for a specific region pair, use Network Topology to examine the exact communication paths between connected workloads. Filter down to individual GKE clusters or peered entities and review latency distributions across the 50th, 95th, and 99th percentiles over time ranges up to six weeks to identify whether the issue stems from specific VM instances, pod communication bottlenecks, or intermediary network hops.
In low-traffic environments, sixty-second probes can cause falsely high RTT readings if the sampling misses the TCP acknowledgment and counts a later data response instead. Additionally, delayed application-layer responses can be misinterpreted as slow network round trips, artificially inflating RTT values. Always check if high RTT levels correspond with actual application-level delays rather than network issues.
Professional Cloud Network Engineer
Prepare and test your skills
Prepare and test your skills
StudyCerts
StudyCerts