Prepare and test your skills
Prepare and test your skills
Network and Security Connectivity Troubleshooting is the practice of separating network failures from permission failures to find the root cause of access problems. The method first checks whether traffic can travel from a source to a destination, then checks whether the identity is allowed to perform the action. In Google Cloud, network checks use Connectivity Tests and flow logs, while permission checks use Policy Troubleshooter.
Connectivity Tests (opens in a new tab) is a static configuration analyzer in Network Intelligence Center that checks connectivity without sending live traffic. Connectivity Tests analyzes the configured path from a source endpoint to a destination endpoint and reports where the path breaks. Because the analysis reads configuration instead of sending packets, it can distinguish a network problem from a permission problem early.
Traffic flows from the source, such as a virtual machine (VM) or subnet, across Virtual Private Cloud (VPC) routes, VPC Network Peering connections, and virtual private network (VPN) tunnels to the destination. The destination can be another VM, a Google API, or an on-premises address. If a firewall rule blocks the path, Connectivity Tests identifies that rule as the root cause, which is especially useful when hybrid connectivity adds extra hops between cloud and on-premises.
VPC Flow Logs is a per-subnet logging feature that records actual network traffic entering and leaving VMs in that subnet. Sampled packets collected during an aggregation interval for a connection are combined into a single flow log entry, and that entry flows from the subnet to Cloud Logging. VPC Flow Logs is enabled or disabled for each VPC subnet, and it generates a large volume of data, so sustained logging should not be left on for development subnets.
Firewall Rules Logging is a per-rule logging feature that audits and verifies the effect of individual firewall rules. Logging is enabled individually for each firewall rule whose connections need review, so an allow rule or deny rule can be checked to see if it functions as intended. When troubleshooting traffic issues, VPC Flow Logs shows whether traffic left or entered a VM through the expected port, while Firewall Rules Logging shows which rule allowed or denied that traffic.
Cloud Audit Logs is the record of who performed which action, where they did it, and when they did it across each project, folder, and organization. The protoPayload field in Cloud Logging stores an AuditLog object for each entry, and entries can be viewed in Cloud Logging or exported to Cloud Storage, BigQuery, or Pub/Sub for longer retention. Viewing Admin Activity logs requires the Logs Viewer role (roles/logging.viewer) or the basic Viewer role (roles/viewer), with least privilege preferred.
Cloud Audit Logs contains three audit log streams:
Configuration drift often causes new access failures, so change detection complements log review. Cloud Asset Inventory keeps a five-week history of asset metadata for supported resources, including Identity and Access Management (IAM) policies, firewall rules, forwarding rules, Google Kubernetes Engine (GKE) namespaces, and role bindings. Real-time notifications flow from Cloud Asset Inventory through Pub/Sub when a resource changes, and a logging sink can filter sensitive calls such as SetIamPolicy to trigger a Cloud Run function for alerting.
Policy Troubleshooter is a Google Cloud console tool that explains why a principal does or does not have permission to access a resource. The IAM part of Policy Troubleshooter compares the effective policy for the principal on the resource and shows which permissions are granted or missing. A task may require many underlying permissions at once, so a user who needs the compute.imageUser role must hold a combination of roles that together include all nine permissions in that role.
Policy Troubleshooter requires three inputs:
IAM Recommender is an additional troubleshooting source when access that previously worked suddenly stops. Recommender runs a daily job that analyzes access log data against permissions granted over the previous 60 days. If a recommendation to remove unused permissions was recently approved and applied, Recommender reveals that change, and the removed permissions can be granted again to restore access.
Troubleshooting across hybrid and multi-cloud environments must account for inconsistent security models, different shared responsibility boundaries, and siloed observability tools. Each cloud provider defines its own controls, service names, compliance obligations, and division of customer versus provider responsibilities. This difference increases the risk of misconfiguration across VPN tunnels, interconnects, and peering connections.
A centralized visibility model avoids manual correlation by collecting network, audit, and security signals in one place. Security Command Center (opens in a new tab) with Security Health Analytics identifies misconfigurations and vulnerabilities in Google Cloud and recommends remediation. Google Security Operations SIEM collects and analyzes security logs from hybrid and multi-cloud environments in one place, which supports consistent detection and response when connectivity spans organization, folder, VPC, and on-premises boundaries.
Each troubleshooting tool answers a different question about the same failed access, so the choice depends on whether configuration, actual traffic, or effective permission evidence is needed. Connectivity Tests validates the intended network path, flow and firewall logs validate what traffic occurred, and Policy Troubleshooter validates what IAM allows. Using them together prevents fixing a firewall when the real problem is a missing permission, or changing IAM when the real problem is a blocked route.
A decision tree that branches from an access failure on whether the cause is network or permission to select Connectivity Tests, flow and firewall logs, or Policy Troubleshooter.
| Tool | What it checks | Use when |
|---|---|---|
| Connectivity Tests | Static configuration of the path from source endpoint to destination endpoint across VPC network, VPC Network Peering, and VPN tunnels | Access may be blocked by routing or a firewall rule and the network versus permission cause is still unknown |
| VPC Flow Logs and Firewall Rules Logging | Actual sampled traffic entering or leaving a VM and the firewall rule that allowed or denied it | Traffic needs verification by port and direction, or a firewall rule needs proof that it works as intended |
| Policy Troubleshooter | Effective IAM permission for a principal on a specific resource | A user or service account is denied an action and the missing underlying permission must be identified |
Exam tip: Policy Troubleshooter requires the underlying permission, not the IAM role name, so enter values like compute.instances.create rather than a role when checking access.
Systematic Diagnostic Procedures and Observability Tools are the SRE-based practices and Google Cloud services used to detect, isolate, and resolve failures. A systematic approach defines clear reliability goals, tracks metrics across system layers, and uses observability insights for continuous improvement. In Google Cloud, this practice is implemented with Google Cloud Observability (opens in a new tab), the integrated set of services to monitor, log, trace, and profile applications.
Reliability goals translate business outcomes into measurable technical targets that guide diagnosis. The process starts by identifying key performance indicators (KPIs) such as revenue influenced by recommendations, customer churn, or user engagement. Each KPI is then mapped to technical metrics such as request success rate, response latency including time to first token (TTFT) and token streaming, rate of irrelevant or harmful responses, and rate of successful task completion.
Service level objectives (SLOs) quantify acceptable reliability from the user's perspective so teams know when behavior counts as a failure. Example SLO targets include:
For training workloads, reliability metrics include model FLOPS utilization (MFU), iterations per second, tokens per second, and tokens per device. Aligning SLOs directly with business needs focuses troubleshooting on the system behavior that most affects users.
Cloud Monitoring is the Google Cloud service that analyzes, monitors, and displays performance data from applications and Google Cloud services. Data flows from infrastructure resources and application frameworks into Cloud Monitoring, where the data is stored, visualized on dashboards, and evaluated by alerting rules. Specialized serving metrics from frameworks such as vLLM or NVIDIA Triton Inference Server flow into Cloud Monitoring through Google Cloud Managed Service for Prometheus (GMP). Teams use pre-built dashboards for Google Cloud services and create custom dashboards for workload-specific indicators.
Cloud Monitoring tracks the four golden signals for applications:
Infrastructure monitoring also tracks processor usage across CPU, GPU, and TPU, memory usage, network throughput and latency, and disk I/O across GKE nodes and Cloud Run instances. When saturation rises while latency and error rate degrade, Cloud Monitoring isolates the anomaly to a resource bottleneck or traffic surge rather than an application logic error.
Cloud Logging is the Google Cloud service that stores, searches, analyzes, monitors, and alerts on log data and events from Google Cloud and AWS. Detailed logs flow from applications and underlying infrastructure into Cloud Logging, where log entries provide context around events and errors that metrics alone cannot explain. This context links a failed request, crashed server, or configuration change to its timestamp, service, and related events.
Cloud Logging isolates service failures by narrowing broad symptoms to specific components and actions. When Cloud Monitoring shows an increased error rate, an operator searches Cloud Logging for error messages, stack traces, and recent configuration operations in the same time window. Because logs preserve the sequence of events leading to a failure, they distinguish between code defects, failed dependencies, and operator actions such as deleted load balancer forwarding rules.
Cloud Trace is the Google Cloud service that finds performance bottlenecks by following requests across distributed microservices. A single user request flows from a frontend service through each downstream microservice, and Cloud Trace records the latency contributed by each span in that path. This end-to-end view is crucial for debugging complex interactions such as agent conversations or multi-component inference pipelines, where delay can originate in any service.
By pinpointing which service adds latency, Cloud Trace directs further investigation to the correct logs and profiles instead of the wrong component. The trace therefore connects a user-visible slowdown to the specific downstream dependency that caused it.
Error Reporting is the Google Cloud service that identifies application errors by aggregating crashes and exceptions. Error data flows from running applications into Error Reporting, which groups repeated errors, counts occurrences, and surfaces new or spiking failures. This grouping lets an operator move from knowing that errors increased to knowing which error signature is responsible and where it occurs in code.
Cloud Profiler is the low-impact CPU and heap profiling service that continuously gathers performance information from production systems. Profiler data reveals bottlenecks within function blocks in application code, which helps optimize resource usage and execution time. While Error Reporting isolates which failure is breaking requests, Cloud Profiler explains which code path is consuming time or memory and causing saturation and latency.
Separating platform incidents from application failures is the first triage step that prevents debugging the wrong layer. The Google Cloud Service Health Dashboard shows the current status of Google Cloud services across all regions, and its history page details each incident with duration, affected zones and regions, affected services, and workarounds. Personalized Service Health (opens in a new tab) shows incidents relevant to a specific project, supports API access on a per-project or per-organization basis, and supports alerts and RSS feeds.
Progressive change deployment reduces the need for this triage by limiting blast radius:
Global resources such as VPC networks and global load balancers require extra control because a single misconfigured forwarding rule can stop the frontend from receiving requests even while backends remain intact. Change review classifies modifications to global resources as high-risk changes that additional reviewers must verify and approve.
Exam tip: If Service Health shows no platform incident, treat unavailability as an application or configuration failure and continue with Cloud Logging, Cloud Monitoring, and Cloud Trace.
Root Cause Analysis Methodologies and Blameless Post-mortems are structured, evidence-driven practices for moving from visible symptoms to the underlying failure that must be fixed. A symptom is what users and alerts observe, such as high latency or failed requests, while the underlying cause is the configuration change, capacity limit, dependency failure, or code defect that produced that symptom. Blameless post-mortems document that chain of evidence without attributing fault to individuals.
Distinguishing symptoms from underlying causes means testing each observation against telemetry and change history until the causal chain is proven. An application can be unavailable even when there is no Google Cloud infrastructure outage, for example when a software update crashes servers or an administrator deletes load balancer forwarding rules. Business-critical operations that target recovery time objective (RTO) and recovery point objective (RPO) must depend only on data plane components that continuously process operations, and must not depend on management plane operations such as the VM-creation API.
A structured investigation asks the same set of questions for every outage: what customers experience, how the outage is detected, what happens to the application and its data, how other applications are affected through cross-dependencies, who performs recovery, and who must be notified within what time period. Answering those questions separates one-time effects from the condition that allowed the failure to spread. When an incident involves a global resource such as a VPC network or global load balancer, that resource deserves extra scrutiny because a single misconfigured forwarding rule can stop the frontend while the backend remains intact.
The observability foundation for diagnosis provides the metrics, traces, and logs needed to prove what failed and in what order. Google Cloud Observability tools work together with Cloud Monitoring to build alerting, dashboards, and investigation workflows, while OpenTelemetry provides an open source standard for capturing metrics, traces, and logs. Google Cloud Managed Service for Prometheus (opens in a new tab) provides a fully managed backend for Prometheus metrics at scale, including AI serving metrics from vLLM and NVIDIA Triton Inference Server.
Diagnosis starts with service-level indicators (SLIs), SLOs, and error budgets that define acceptable reliability from the user's perspective. Example SLO targets include 99.9% of API calls returning success, 95th percentile inference latency below 300 ms, time to first token below 500 ms for 99% of requests, and harmful output below 0.1%. Monitoring then tracks the four golden signals across system layers:
Cloud Logging collects detailed application and infrastructure logs that give context around events, Cloud Trace pinpoints latency and follows request flows across microservices, and Cloud Profiler identifies bottlenecks inside function blocks. Together, infrastructure metrics for processor usage, memory usage, network throughput and latency, and disk I/O connect a user-visible symptom to the saturated resource or slow dependency that caused it.
Structured incident response procedures ensure that evidence is collected quickly, in order, and by the right owners during an incident. A well-established procedure defines roles for incident commanders, investigators, communicators, and technical experts so response stays coordinated. Communication protocols and escalation paths ensure information is shared promptly, documented procedures in a runbook or playbook guide the work, and regular training prepares teams to respond effectively.
The runbook moves through an ordered workflow:
Two complementary sources help determine whether Google Cloud infrastructure is part of the cause. The Service Health Dashboard shows current service status across all regions plus incident history with duration, affected zones and regions, affected services, and workarounds. Personalized Service Health (opens in a new tab) shows incidents relevant to a specific project or organization, supports API access and alerts, and can deliver programmatic updates through RSS feeds.
Exam tip: Use the Service Health Dashboard for global Google Cloud status and history, and use Personalized Service Health for incidents that affect your specific project or organization.
A blameless post-mortem, also called an after action report, is a written record that explains what happened, why it happened, and what will change so it does not happen again. The post-mortem focuses on system design, safeguards, and process gaps rather than individual fault, which encourages complete reporting of timeline, decisions, and contributing factors. Findings flow from the same evidence used during response: SLO breaches, golden-signal charts, log and trace excerpts, change records, and Service Health history.
Prevention depends on testing and controlled change after the post-mortem is complete. Regular performance and load testing, stress testing to system limits, chaos engineering, and automated unit, integration, and end-to-end tests in CI/CD pipelines validate fixes before they reach production. Infrastructure changes are deployed progressively: first change a few VMs in one zone, then monitor effects, revert quickly if issues appear, diagnose and resolve, and only after verification deploy gradually across all infrastructure. Changes to global resources receive rigorous review as high-risk changes, and capacity data from testing informs planning so constraints do not recreate the original bottleneck.
Connectivity Tests validates the static configuration of the path from a source endpoint to a destination endpoint across Virtual Private Cloud (VPC) network, Virtual Private Cloud (VPC) Network Peering, and virtual private network (VPN) tunnels without sending live traffic. Virtual Private Cloud (VPC) Flow Logs and Firewall Rules Logging validate actual sampled traffic entering or leaving a virtual machine (VM) and the firewall rule that allowed or denied it. Use Connectivity Tests when access may be blocked by routing or a firewall rule and the network versus permission cause is still unknown, and use flow and firewall logs when traffic needs verification by port and direction or a rule needs proof that it works as intended.
Policy Troubleshooter requires a principal, a permission, and a resource. The principal is the user, service account, or group that requested access, the permission is the underlying Identity and Access Management (IAM) permission to check rather than the role name, and the resource is the specific project, folder, organization, or resource on which access was requested.
Virtual Private Cloud (VPC) Flow Logs is a per-subnet logging feature that records actual sampled network traffic entering and leaving virtual machines (VMs) in that subnet. Firewall Rules Logging is a per-rule logging feature that audits and verifies the effect of an individual allow or deny firewall rule. When troubleshooting, flow logs show whether traffic left or entered a virtual machine (VM) through the expected port, while firewall logging shows which rule allowed or denied that traffic.
An e-commerce platform running a microservices architecture on Google Kubernetes Engine (GKE) with Cloud Service Mesh experiences intermittent HTTP 500 errors and latency spikes during peak shopping periods.
The operations team must determine whether the failures originate from proxy routing issues or unhandled application code exceptions, while also identifying latency bottlenecks across downstream microservice calls.
Which systematic diagnostic procedure should the team use to isolate the root cause?
Last updated