Troubleshooting Infrastructure Failures in Google Cloud

Persistent storage and data availability problems happen when zonal or regional storage cannot serve reads and writes because a dependency, replication path, or capacity limit has failed. In Google Cloud, Cloud Storage (opens in a new tab), Persistent Disk, Filestore, and managed databases such as Cloud SQL, Spanner, and Bigtable stay available only when their dependencies on encryption, replication, networking, and compute stay healthy. Troubleshooting means tracing from the failed volume, bucket, or database to the dependency that blocks I/O and then restoring service from a replica, snapshot, or standby region.

1.1Storage Dependencies That Drive Availability Failures

Storage dependencies are the key, network, platform, and identity services that must stay healthy for a disk, bucket, or database to remain reachable. Persistent Disk I/O depends on lower-level Compute Engine services such as Borg and Colossus, so a cluster-level control plane or storage fabric problem can stall virtual machine (VM) I/O even when the VM appears running. Spanner, Bigtable, and Cloud Storage depend on encryption and key management for Cloud Key Management Service (Cloud KMS), Cloud External Key Manager (Cloud EKM), and Google-owned keys, on internal logging and auditing of data access, and on internal replication services when data is expected across regions.

All services also share common dependencies that can mimic a storage outage. Access to Google Cloud APIs depends on Domain Name System (DNS), globally distributed load balancers, and points of presence, while global configuration such as Identity and Access Management (IAM) policies is stored in replicated databases. Identity data plane failures for authentication and authorization can therefore block access to buckets, disks, and databases without any loss of stored bytes. When diagnosing, first separate permission and connectivity failures from true durability failures by checking identity, DNS, firewall, and cross-region network paths before assuming data is lost.

1.2Cloud Storage Replication and Data Protection Options

Cloud Storage availability is controlled by bucket location type and by object-level protection. Data in regional buckets is replicated synchronously across multiple zones within one region, which protects against a zonal failure but not a regional loss. Data in dual-region and multi-region buckets is replicated across geographic regions, with dual-region buckets providing geo-redundancy for business continuity and multi-region buckets storing data within the geographic boundary of the multi-region. Dual-region objects can optionally use turbo replication for faster cross-region replication with a lower recovery point objective (RPO).

Protection against deletion and corruption uses different controls than replication:

• Soft Delete preserves deleted and overwritten objects for a specified period to allow recovery from accidental deletion or faulty pipelines.
• Object holds prevent deletion or updates to objects to enforce retention during investigation or compliance holds.
• Storage Transfer Service imports data from Amazon S3, Azure Blob Storage, or on-premises sources into Cloud Storage, backs up data from other providers to a bucket, and moves data from a dual-region or multi-region bucket to a regional bucket to lower backup storage cost.

When using regional or zonal storage, replicate data to another region or snapshot it to a multi-regional resource for disaster recovery (opens in a new tab). That rule applies directly to Cloud Storage planning: keep primary data where latency and locality require it, but maintain a geographically separate copy that survives extended loss of an entire region.

1.3Persistent Disk and Filestore Resilience

Persistent Disk resilience depends on choosing zonal versus regional disks and on snapshot strategy. Zonal disks live in one zone and require replication or snapshots elsewhere to survive zonal failure, while regional persistent disks automatically replicate data to two zones in a region so I/O can continue if one zone becomes unavailable. Compute Engine managed instance groups can distribute VM instances across multiple zones in a region and manage them as a logical unit, which pairs with regional disks to keep both compute and attached state available. For recovery, launch replacement VMs from pre-created snapshots and keep stopped standby instances when needed, since stopped instances incur only storage costs and substantially reduce the cost of cold standby.

Filestore resilience depends on tier because tiers have different failure domains. Filestore Basic and Zonal tiers are zonal resources and do not support replication across zones, so a zonal outage can make those Network File System (NFS) file servers used by Compute Engine instances or Google Kubernetes Engine (GKE) clusters unavailable. Filestore Enterprise tier instances are regional resources and are recommended to increase resiliency of the file environment. If an application requires file data to survive zone loss without manual restore, choose Enterprise tier; if it can tolerate restore from backup, a zonal tier with snapshots to multi-regional Cloud Storage can lower cost at the expense of higher recovery time objective (RTO).

1.4Managed Database Continuity and Replication Failures

Managed database continuity depends on Compute Engine plus continuous data replication (opens in a new tab) to a standby. Managed and hosted services like GKE and Cloud SQL depend on Compute Engine and on Container Registry or Artifact Registry for VM images, so image unavailability or zonal compute shortage can block database failover even when data replicas are healthy. In the on-premises to Google Cloud pattern, the database is pre-provisioned as a VM-based database or as a managed database like Cloud SQL with continuous replication, web and application VMs are launched from snapshots after the failure event, and DNS is repointed to the Cloud Load Balancing external IP address. The full invoke, provision, and reroute sequence can be manual or automatic depending on RTO and policy, and infrastructure as code with Terraform applied through Cloud Build speeds and automates provisioning.

Replication failures also include logical hazards such as degraded throughput and split-brain writes. Minimize synchronous dependencies between systems running in different environments because synchronous cross-environment calls slow performance and decrease overall availability, and account for cross-regional and cross-zonal bandwidth when replicas, snapshots, and backup traffic cross boundaries. If data is replicated bidirectionally and connectivity is lost, both environments can conclude the other is unavailable and accept conflicting writes. Avoid that split-brain state by using a third computing environment to establish quorum before modifying data or by allowing conflicting modifications to be reconciled after connectivity returns, and with SQL databases make the original primary inaccessible before clients start using the new primary.

1.5Recovery Procedures for Durability and Service Continuity

Recovery procedures for durability combine standby pattern, backup location, and end-to-end restore testing. Cold, warm, and hot disaster recovery patterns (opens in a new tab) define readiness: cold has no running standby and must be built after failure, warm has a standby kit such as snapshots and stopped instances that must be started, and hot runs active replicas such as multi-regional storage or continuously replicated Cloud SQL that continue with little interruption. Design to recovery goals by matching RTO and RPO to pattern, since smaller RTO and RPO values increase cost and operational complexity. Historical compliance data with large RTO can use cold backup to Cloud Storage, notification systems can use warm standby, and online user-facing services need hot replication.

Durability work is not complete until restore, access, and audit paths are proven. Maintain more than one data recovery path in case the primary connection to Google Cloud is unavailable, automate provisioning with Terraform and Infrastructure Manager, and monitor with Google Cloud Observability using uptime checks, alerts, logging, and real-time configuration notifications through Pub/Sub. Treat recovered data like production data by applying the same permissions, encryption, and audit controls, knowing where backups live and who can restore them, backfilling logs collected in the recovery environment into the production archive, and regularly testing permissions, penetration resistance, and failover when the normal access path does not work.

Infrastructure resource constraints and scaling failures occur when compute, memory, or network demand exceeds planned capacity, quota limits, or autoscaling policy. In Google Cloud, Compute Engine (opens in a new tab), Google Kubernetes Engine (GKE), Cloud Run, Bigtable, and Managed Service for Apache Spark all depend on planned capacity, quota headroom, and autoscaling policy to match resources to load. Mitigation combines capacity planning, quota and reservation management, continuous monitoring, and validation through load testing.

2.1Capacity Planning to Prevent Exhaustion

Capacity planning (opens in a new tab) is the ordered practice of forecasting load, sizing resources, adding redundancy, and verifying efficiency before exhaustion occurs. Resource efficiency is the relationship between application load and central processing unit (CPU) and memory consumed, and efficiency can deteriorate when load is exceptionally high or as code changes over time. A disciplined planning sequence prevents compute, memory, and persistent disk bottlenecks:

1. Forecast application load: Estimate users and request rate using historical trends, seasonal variations, spikes during special events, and growth from business changes like expansion to new geographies.
2. Estimate capacity requirements: Translate forecast load into quantity, type, and size of resources, such as managed instance group (MIG) size, VM machine type, and number, type, and size of persistent disks, using the Google Cloud Pricing Calculator for cost modeling.
3. Plan adequate redundancy: Provision beyond the minimum needed to handle forecast load, such as N+1 redundancy where every component has at least one redundant component beyond the minimum.
4. Benchmark the application: Run load tests under both normal and peak conditions to measure how CPU and memory grow with load, then repeat at regular intervals.
5. Secure quotas and reservations: Verify quota headroom and create assured capacity before deployment.
6. Monitor utilization and reassess: Track utilization after deployment to remove idle resources and periodically reassess for changes in behavior, performance objectives, user load, and budget.

Static redundancy keeps an application available during location outages and load spikes, but during low load the extra capacity remains unused or under-utilized. Autoscaling resolves that tradeoff by provisioning additional resources automatically when load rises and removing unused resources when load falls.

2.2Quotas and Reserved Capacity as Hard Limits

Google Cloud quotas are per-project limits that control consumption and can block creation or scaling even when autoscaling policy is correct. Resource quotas set the maximum resources that can be created, such as the number of regional GKE clusters in a region, while rate quotas limit the number of API requests that can be sent to a service in a specific period. Quotas can be zonal, regional, or global, so a scale-out failure in one zone or region may be a quota failure rather than a lack of physical capacity. When forecast capacity exceeds current limits, request a quota adjustment before peak load occurs.

Compute Engine reservations provide assured capacity in a specific zone for a specified number of VMs of a chosen machine type. A reservation flows from the reservation holder to matching VM creation requests in that zone, guaranteeing placement when on-demand capacity would otherwise be uncertain. A reservation can be specific to a project or shared across multiple projects. Choose reservations when startup latency or availability during a peak event cannot depend on spare zone capacity, and choose quota increases when the blocker is the project limit rather than physical assurance.

2.3Detecting Compute, Memory, and Network Exhaustion

Resource exhaustion appears first in utilization and performance signals collected by Cloud Monitoring (opens in a new tab) and analyzed with tools like BigQuery and Active Assist (Recommender). Monitoring flows from instrumented resources to Cloud Monitoring metrics, then to dashboards and alerts that notify teams before saturation causes errors. Track system health indicators that align directly with exhaustion types:

• CPU utilization for compute saturation and throttling
• Memory usage for pressure, evictions, and out-of-memory failures
• Network traffic for bandwidth saturation and cross-zone or cross-region congestion
• Disk I/O for storage throughput and persistent disk bottlenecks
• Application response times and business-specific metrics for user-visible impact of underlying constraints

Set alerts on critical metrics with thresholds tuned to minimize alert fatigue while ensuring timely response to significant issues. Use Active Assist recommendations to identify underutilized resources, right-sizing opportunities, and architectural risks, and use historical usage patterns and growth projections to forecast future demand including seasonal variations, product launches, marketing campaigns, and failover needs for primary and disaster recovery systems.

2.4How Autoscaling Works and Why It Fails

Autoscaling is the policy-driven adjustment of capacity based on workload signals, and each Google Cloud service scales a different unit in response to different signals. Compute Engine MIGs manage groups of VMs as a single entity and scale stateless applications using autoscaling policies that define minimum and maximum VM counts and the conditions that trigger scaling, such as adding VMs when CPU utilization reaches a threshold and removing VMs when it drops below a different threshold. GKE cluster autoscaling resizes infrastructure and workloads together, while Cloud Run and Cloud Run functions scale serverless capacity automatically with incoming requests. Bigtable and Managed Service for Apache Spark scale data and batch resources to target utilization and efficient execution.

Autoscaling failures typically fall into three root causes. First, velocity failure occurs when event traffic rises faster than new VMs, nodes, or instances can start, so availability drops even though policy is correct. Second, configuration failure occurs when minimum, maximum, or utilization targets are too low, too restrictive, or based on the wrong metric, so the autoscaler never triggers or hits its ceiling early. Third, dependency failure occurs when quota limits, zonal capacity shortages, or unhealthy backends prevent otherwise correct scale-out from serving traffic. For example, an ecommerce shopping cart that must process 99.9% of orders within 200 milliseconds after confirmation needs redundant compute and storage during high load, but without sufficient maximums, quota, and healthy backends, autoscaling cannot sustain that objective.

2.5Load Balancing and Health Checks to Contain Failures

Load balancing improves reliability by routing traffic only to available resources and ensuring individual resources are not overloaded. Traffic flows from external clients or from an upstream tier to the load balancer, then from the load balancer to only the healthy backends in its backend pool. Configure load balancing for traffic between application tiers as well as from external clients, such as using an internal load balancer for reliable communication between the web and application tiers in a 3-tier stack. Choose a global load balancer for multi-region distribution and failover, and choose a regional load balancer when backends sit in a single region and must remain resilient to zone outages.

Load-balancing health checks prevent the autoscaler from masking application failure with more capacity. Even when infrastructure exists and all VMs hosting an application are in the RUNNING state, the application software on some VMs may have crashed. Health checks probe backend responsiveness and the load balancer depends on those results to exclude unresponsive backends from traffic. When latency rises but CPU and autoscaler behavior look normal, compare VM state against health-check state to distinguish resource exhaustion from application-level failure on healthy infrastructure.

2.6Validating Scale with Load Testing and Pre-Warming

Load testing (opens in a new tab) determines whether the system will scale successfully in production and finds bottlenecks that prevent scaling. Testing should occur three to five months before a peak event on critical projects and workloads to simulate peak traffic. After each test, evaluate results, use Capacity Planner to visualize usage and forecast data, and request quota increases as necessary. Investigate quota and cost implications before stressing services and consider creating Cloud Billing budget alerts to monitor expense for the most stressed services.

Because autoscaling might not catch up with the velocity of event traffic, manually scale up and pre-warm resources ahead of time, including VMs, pre-loaded caches, and serverless components to prevent cold starts. This pre-warming dependency runs first from operations to the resource pool, then autoscaling takes over from the warmed baseline to handle fluctuations during the event. If load testing reveals sustained saturation, adjust machine types, MIG minimums and maximums, GKE node pools and Pod autoscaling targets, or Bigtable target utilization before the event rather than relying on reactive scaling alone.

Connectivity and network configuration troubleshooting is the systematic isolation of blocked or misrouted traffic across Virtual Private Cloud (VPC) networks, firewalls, load balancers, DNS, and hybrid links. Traffic flows from the source VM, container, or on-premises host through VPC routes, firewall policy evaluation, Network Address Translation (NAT) or Private Service Connect endpoints, and then to the destination service or Google API. A failure at any boundary appears as timeouts, connection refusals, unhealthy backends, or DNS resolution errors, so resolution depends on correlating configuration state with telemetry from logs and active diagnostics.

3.1VPC Connectivity and Private Access to Google Services

Virtual Private Cloud (VPC) (opens in a new tab) connectivity depends on the correct network, subnet, routes, and private-access settings for workloads that use only internal IP addresses. Resources without external addresses remain isolated from the public internet but can still reach Google APIs when Private Google Access is enabled at the subnet level, and Private Service Connect (PSC) provides private consumer endpoints for Google APIs and for services published in other VPCs or on-premises networks. When PSC for Google APIs is used, DNS records for domains such as storage.googleapis.com must resolve to private IP addresses inside the VPC network, and custom DNS servers must forward Google domains correctly. If resolution returns public addresses or HTTPS probes such as curl -v https://storage.googleapis.com fail from a test VM in the same subnet with the same service account, the fault lies in subnet configuration, PSC endpoint state, or missing Private Google Access rather than in the application.

3.2Firewall Rules and Policy Troubleshooting

Cloud Next Generation Firewall (Cloud NGFW) controls which traffic is allowed between VPC resources through distributed firewall rules and hierarchical firewall policies applied at the organization, folder, and VPC levels. Each firewall rule is evaluated by priority where lower numbers win, so a general low-priority allow can shadow a specific higher-numbered deny unless specific rules are numbered lower than general rules. Firewall Rules Logging must be enabled individually for each rule whose connections need auditing, and logs flow from the firewall enforcement point to Cloud Logging where allow and deny decisions can be verified with Firewall Insights in Network Intelligence Center.

The following priority scheme prevents shadowing and preserves room for emergency response:

Hierarchical policies enforce consistent baselines across projects, including least-privilege limits that restrict each rule to only the protocols and ports each workload needs. When connectivity fails after a policy change, compare the effective hierarchy from organization to VPC, check for shadowed rules, and confirm that logging is enabled on the suspect rule before changing priorities.

3.3Load Balancer Health Checks and Backend Reachability

Cloud Load Balancing distributes traffic to backends only when its probers can reach those backends, so a backend marked unhealthy is usually a firewall or route problem rather than an application outage. Health-check traffic flows from Google probers to each backend VM or endpoint, and the firewall policy must explicitly allow ingress from the well-known prober ranges or all probes will be dropped. For most load balancers, including internal TCP/UDP, internal HTTP(S), external TCP proxy, external SSL proxy, and external HTTP(S), the policy must allow ingress from 35.191.0.0/16 and 130.211.0.0/22 for TCP ports 80 and 443. For Network Load Balancing with legacy health checks, the policy must instead allow ingress from 35.191.0.0/16, 209.85.152.0/22, and 209.85.204.0/22.

When backends flap between healthy and unhealthy, verify that the hierarchical policy contains both the IAP and health-check allows, that backend firewall rules permit the probe ports, and that return routes from the backend to the prober are not forced asymmetrically through a network virtual appliance that drops the reply.

3.4Cloud DNS Resolution

Cloud DNS serves public zones, private zones, and forwarding policies that control how queries from a VPC are answered based on source and destination. A separate built-in service, internal DNS, automatically resolves internal VM names inside the VPC without custom servers, while Cloud DNS policies can forward queries to existing on-premises servers or override private responses by query name. Resolution for Google APIs depends on the access path: with Private Google Access or PSC, queries for storage.googleapis.com, bigquery.googleapis.com, and dataproc.googleapis.com must resolve to private addresses, and a response with public addresses indicates a missing private zone, incorrect forwarding rule, or PSC endpoint DNS misconfiguration. To isolate DNS from routing, first run nslookup storage.googleapis.com from a proxy VM in the workload subnet, then run curl -v to the same name and ping 8.8.8.8 or curl -v https://example.com to test whether IP routing and Cloud NAT egress work when name resolution succeeds.

3.5Hybrid Connectivity Through VPN and Interconnect

Hybrid connectivity extends a VPC to on-premises or other clouds through Cloud VPN (opens in a new tab), Cloud Interconnect with Dedicated or Partner attachments, and Cloud Router which exchanges dynamic routes with peer routers using Border Gateway Protocol (BGP). Traffic flows from the VPC subnet through Cloud Router-learned routes to virtual local area network (VLAN) attachments — normally two attachments, one per edge availability zone — and then across the Interconnect or IPsec VPN tunnel to the peer network router. Individual VPN tunnels have bandwidth limits, so insufficient tunnel count causes congestion and packet loss under load, while a missing or withdrawn BGP route causes complete loss of the remote prefix. For 99.99% availability, Cloud Interconnect normally must connect into two different Google Cloud regions from different metros, even when VMs run in only one region.

Convergence and symmetry add further nuance for advanced troubleshooting. Bidirectional Forwarding Detection (BFD) for BGP speeds detection of path failures, but BFD is not supported on BGP sessions for Cloud VPN tunnels or for Router appliance spokes, so those paths depend on slower BGP timers. Designs that require symmetric routing through centralized appliances cannot simply add nodes; they depend on BGP attributes for single-active forwarding, source NAT on the appliance, vendor session synchronization, or BGP route policies for active-standby per flow. Increasing the maximum transmission unit (MTU) can improve throughput, but mismatched MTU between VPC, Interconnect, VPN, and on-premises devices causes fragmentation or black-holing that appears as large-packet failures while small packets succeed.

3.6Telemetry and Diagnostic Tools for Root Cause Analysis

Network telemetry and diagnostics correlate live traffic behavior with static configuration to distinguish network denials from permission denials. The core signals flow from enforcement points and routers to observability services that preserve what Connectivity Tests alone cannot see over time.

• VPC Flow Logs record sampled traffic patterns between VPC endpoints to reveal rejected flows, cross-zone or cross-region volume, and unexpected egress.
• Firewall Rules Logging records allow and deny evaluations for enabled rules to prove whether a specific rule blocked the flow.
• Cloud NAT logging records address translations, with ERRORS_ONLY as the minimum and ALL required to see successful translations.
• Cloud Logging and Cloud Monitoring aggregate audit, system, and workload logs, including Spark driver and executor ERROR or WARNING messages for timeouts and API failures.
• Firewall Insights analyzes rule usage, shadowing, and overly permissive rules to optimize policy without breaking connectivity.

Connectivity Tests in Network Intelligence Center is a static configuration analyzer that simulates a path from a source to a destination across VPC networks, VPC Network Peering, and VPN tunnels to on-premises networks. The test depends on the Network Management API being enabled, and it identifies the blocking construct such as a firewall rule, missing route, or tunnel misconfiguration before packets are sent. Use it in sequence with live evidence to reach root cause:

1. Identify configuration: Review the workload networkUri and subnetUri, VPC routes, firewall hierarchy, and DNS policy for the failing subnet.
2. Test live from a proxy VM: Launch a test Compute Engine VM in the same subnet with the same service account, then run nslookup, curl -v to Google APIs, and ping or curl to the internet in order.
3. Simulate the path: Run Connectivity Tests from the source subnet to the destination endpoint, such as a Google API, peered VPC address, or on-premises prefix.
4. Confirm in logs: Review Firewall Rules Logging, VPC Flow Logs, Cloud NAT logs, and workload logs in Cloud Logging to match the simulated block with observed denies, timeouts, or translation errors.

Key Facts

• Regional persistent disks replicate data to two zones in a region to survive zonal failure, while Filestore Enterprise tier is regional and Filestore Basic and Zonal tiers are zonal without cross-zone replication.
• Dual-region Cloud Storage buckets replicate across two geographic regions with optional turbo replication for lower RPO, while regional buckets replicate only across zones in one region.
• Google Cloud quotas are per-project zonal, regional, or global limits that can block scale-out, while Compute Engine reservations provide assured capacity in a specific zone for a chosen machine type.
• Cloud Load Balancing sends traffic only to healthy backends, and most load balancers require firewall ingress allows from 35.191.0.0/16 and 130.211.0.0/22 for health-check probes.
• Connectivity Tests simulates a path from source to destination to identify firewall, route, or tunnel misconfigurations, while Firewall Rules Logging and VPC Flow Logs confirm observed denies and rejected flows.