Troubleshooting Infrastructure, CI/CD and App Failures

Persistent storage, capacity, and network failures in Google Cloud share one troubleshooting rule: trace from the failed resource to the dependency that blocks it before assuming data is lost. Traffic flows from virtual machine (VM) instances, containers, or on-premises hosts through Virtual Private Cloud (VPC) routes and firewall enforcement to Persistent Disk (opens in a new tab) volumes, buckets, or databases, while identity, Domain Name System (DNS), and replication services must stay healthy on both sides of every trust boundary. When reads, writes, or scale-out stall, the cause is usually a failed key, route, quota, or replica path rather than destroyed bytes.

1.1Storage Dependencies That Control Availability

Storage dependencies are the encryption, network, platform, and identity services that must stay healthy for a disk, bucket, or database to serve input/output (I/O). Persistent Disk I/O depends on lower-level Compute Engine services such as Borg and Colossus, so a control-plane or storage-fabric problem can stall VM I/O while the VM still appears running. Spanner, Bigtable, and Cloud Storage (opens in a new tab) depend on Cloud Key Management Service (Cloud KMS), Cloud External Key Manager (Cloud EKM), or Google-owned keys, plus internal logging, auditing, and cross-region replication. All access also depends on DNS, globally distributed load balancers, points of presence, and Identity and Access Management (IAM) identity data planes, so first separate permission and connectivity failures from durability failures by checking IAM, DNS, firewall, and cross-region paths.

1.2Cloud Storage Replication and Disk Resilience

Cloud Storage availability and Persistent Disk resilience are controlled by choosing zonal versus regional failure domains and by keeping a separate recovery copy. Data in regional Cloud Storage buckets is replicated synchronously across multiple zones in one region and survives zonal failure but not regional loss, while dual-region buckets replicate across two geographic regions for geo-redundancy and multi-region buckets replicate within a geographic boundary. Dual-region objects can use turbo replication for faster cross-region replication with a lower recovery point objective (RPO). Protection against deletion uses Soft Delete (opens in a new tab) to preserve deleted objects, object holds to block deletion, and Storage Transfer Service to import from Amazon S3, Azure Blob Storage, or on-premises and to move data to regional buckets for cheaper backup storage.

Persistent Disk and Filestore follow the same zonal-versus-regional tradeoff. Zonal persistent disks live in one zone and need snapshots or replication elsewhere, while regional persistent disks replicate to two zones in a region so I/O continues if one zone fails. Compute Engine managed instance groups distribute VMs across zones as one logical unit and pair with regional disks to keep compute and state available. Filestore Basic and Zonal tiers are zonal Network File System (NFS) servers without cross-zone replication, while Filestore Enterprise tier is regional and recommended when file data must survive zone loss without manual restore.

1.3Database Continuity, Split-Brain Prevention, and Recovery Patterns

Managed database continuity depends on Compute Engine, container images, and continuous replication to a standby that can take over within recovery targets. Managed services depend on Compute Engine and on Container Registry or Artifact Registry for VM images, so missing images or zonal compute shortage can block failover even when data replicas are healthy. In failover, the database is pre-provisioned as a VM-based or managed database such as Cloud SQL with continuous replication, web VMs launch from snapshots after the event, and DNS repoints to the Cloud Load Balancing external address. The invoke, provision, and reroute sequence can be manual or automatic, and Terraform applied through Cloud Build with Infrastructure Manager speeds provisioning.

Replication also fails logically through slow synchronous calls and conflicting writes. Minimize synchronous dependencies across environments because they slow throughput and lower availability, and account for cross-zone and cross-region bandwidth for replicas and backups. If bidirectional replication loses connectivity, both sides can accept conflicting writes in a split-brain state. Avoid it with a third environment for quorum before writes or by reconciling conflicts after reconnect, and with SQL databases make the original primary inaccessible before clients use the new primary. Recovery readiness follows cold, warm, and hot patterns: cold builds after failure, warm starts snapshots and stopped instances that incur only storage cost, and hot continues on active replicas such as multi-regional storage. Smaller recovery time objective (RTO) and RPO values increase cost, so match cold to large-RTO compliance data, warm to notification systems, and hot to user-facing services.

1.4Capacity Planning, Quotas, and Autoscaling Behavior

Capacity, quota, and autoscaling failures occur when demand exceeds forecast resources, project limits, or policy bounds. Capacity planning is the ordered practice of forecasting load, sizing resources, adding redundancy, and verifying efficiency before exhaustion. Resource efficiency relates application load to central processing unit (CPU) and memory consumed and can degrade at high load or as code changes:

1. Forecast application load: Estimate users and request rate from history, seasons, events, and growth such as new geographies.
2. Estimate capacity requirements: Translate load into managed instance group (MIG) size, VM machine type, and persistent disk count and size, using the Google Cloud Pricing Calculator.
3. Plan adequate redundancy: Provision beyond minimum, such as N+1 redundancy with one extra component beyond minimum.
4. Benchmark the application: Load-test under normal and peak conditions and repeat regularly.
5. Secure quotas and reservations: Verify quota headroom and create assured capacity before deployment.
6. Monitor utilization and reassess: Track use with Cloud Monitoring (opens in a new tab) and Active Assist, then remove idle resources and reassess.

Google Cloud quotas are per-project zonal, regional, or global limits that block creation even when policy is correct. Resource quotas limit creatable resources such as regional Google Kubernetes Engine (GKE) clusters, while rate quotas limit API requests per period. Compute Engine reservations provide assured capacity in a specific zone for a chosen machine type and can be project-specific or shared. Autoscaling then adjusts within those limits: Compute Engine MIGs scale VMs on CPU or custom metrics between minimum and maximum, GKE Cluster Autoscaler adds nodes on Pod demand while Horizontal Pod Autoscaler changes Pod replicas and Vertical Pod Autoscaler tunes requests, Cloud Run scales container instances with traffic including scale to zero, and Bigtable scales nodes to target utilization. Failures come from velocity when traffic rises faster than startup, configuration when minimums or maximums are too restrictive, and dependencies when quota, zonal capacity, or unhealthy backends block scale-out. Load balancing contains the impact by sending traffic only from the load balancer to healthy backends, with global load balancers for multi-region failover and internal load balancers between tiers such as web to application.

1.5VPC Access, Firewall Priority, DNS, and Hybrid Path Diagnostics

Connectivity troubleshooting isolates blocked traffic across VPC routes, Cloud Next Generation Firewall (Cloud NGFW), load-balancer health checks, Cloud DNS, and hybrid links. Traffic flows from the source VM or container through VPC routes and firewall evaluation, then through Network Address Translation (NAT) or Private Service Connect (PSC) endpoints to the destination or Google API. Resources without external addresses can still reach Google APIs when Private Google Access is enabled on the subnet, while PSC provides private consumer endpoints for Google APIs and cross-VPC services. With PSC for Google APIs, DNS for names such as storage.googleapis.com must resolve to private addresses, so test with nslookup, then curl -v https://storage.googleapis.com, then ping 8.8.8.8 to separate DNS failure from routing failure.

Firewall and health-check failures mimic application outages. Cloud NGFW rules are evaluated by priority where lower numbers win, so number specific rules lower than general rules to avoid shadowing. Firewall Rules Logging must be enabled per rule, with logs flowing to Cloud Logging for verification with Firewall Insights in Network Intelligence Center. Hierarchical policies at organization, folder, and VPC levels enforce least-privilege baselines. Health-check traffic flows from Google probers to backends, so the firewall must allow ingress from 35.191.0.0/16 and 130.211.0.0/22 for most load balancers on TCP ports 80 and 443, and from 35.191.0.0/16, 209.85.152.0/22, and 209.85.204.0/22 for Network Load Balancing with legacy checks. Identity-Aware Proxy (IAP) TCP forwarding uses the distinct range 35.235.240.0/20 for ports 22 and 3389.

Hybrid connectivity extends the VPC through Cloud VPN, Cloud Interconnect with Dedicated or Partner attachments, and Cloud Router exchanging Border Gateway Protocol (BGP) routes with peer routers. Traffic flows from VPC subnets through Cloud Router-learned routes to virtual local area network (VLAN) attachments, normally two attachments across edge zones, then across tunnels to the peer router. Individual VPN tunnels have bandwidth limits, so add tunnels to avoid congestion, while missing BGP routes cause full prefix loss. For 99.99% availability, Interconnect normally enters two regions from different metros. Bidirectional Forwarding Detection (BFD) speeds failure detection but is not supported on Cloud VPN tunnels or Router appliance spokes. Correlate static checks with live telemetry using Connectivity Tests to simulate the path from source to destination, plus VPC Flow Logs for rejected flows, Cloud NAT logging with ERRORS_ONLY minimum and ALL for successes, and workload logs for timeouts.

Continuous integration and continuous delivery (CI/CD) stalls in Google Cloud usually trace to manifests, connectivity, scan gates, or build identity rather than to application code. A release flows from container images built by continuous integration (CI) through Cloud Deploy (opens in a new tab) rendering and promotion into Google Kubernetes Engine (GKE) or Cloud Run targets, while Artifact Registry, Artifact Analysis, and Cloud Build gate each handoff on vulnerability metadata and permissions. When promotion stops, first identify whether rendering, target reachability, scanning, or execution identity blocked the chain.

2.1Skaffold Rendering and Release Promotion Through Targets

Skaffold configuration tells Cloud Deploy which manifests to render and how to deploy them to GKE. A minimal skaffold.yaml file must start with a supported apiVersion such as skaffold/v4beta7, declare manifests under manifests.rawYaml, and declare the deployer under deploy.kubectl. Cloud Deploy calls skaffold render to fill templates and bind image tags, then calls skaffold apply to deploy the rendered output. If manifests are missing they must be created first, and templated manifests must be managed with Kustomize, Helm, or kpt so rendering has complete values. A Cloud Deploy-generated skaffold.yaml file fits only onboarding or demonstration and must not be used for production, which needs an owned file with explicit manifest and deploy sections.

Release creation moves built artifacts through the ordered targets of a delivery pipeline. Prerequisites that must all exist are the execution service account with Cloud Deploy IAM roles, the registered delivery pipeline and targets matched to GKE, GKE attached clusters, Cloud Run, or custom targets, container images from Cloud Build or Jenkins plus Kubernetes manifests or Cloud Run service YAML, and the owned skaffold.yaml file. The workflow runs in order:

1. Build artifacts: Run CI to build images where Cloud Deploy can reference them.
2. Create release: Run gcloud deploy releases create RELEASE_NAME --delivery-pipeline=PIPELINE_NAME --region=REGION from the Skaffold directory or with --source= pointing there.
3. Progress targets: Advance the release through the pipeline, with rendering completed before each deployment.

The command packages the current directory and subdirectories into a tar file, so running from home or root uploads unwanted files. RELEASE_NAME must be unique per pipeline, and dynamic names can use $DATE or $TIME where rel-$TIME at 3:07pm UTC becomes rel-1507.

2.2Private Cluster Connectivity and Execution Environments

Target connectivity controls whether the Cloud Deploy control plane and execution environment can reach the GKE control plane to apply manifests. For clusters accessed only through an HTTP proxy, the target adds proxyUrl to the gke stanza alongside the cluster resource name, and deployment traffic flows from Cloud Deploy through that proxy URL. A missing or wrong proxyUrl value appears as deployment-stage failure even when rendering succeeded. For private GKE clusters, three paths connect to the isolated control plane: a DNS-based endpoint, a VPC network with a private worker pool, or GKE attached clusters with Connect gateway.

The DNS-based endpoint is simplest because nodes and Pods stay private while the control plane exposes a DNS name. Enable the DNS-based endpoint on the cluster and set dnsEndpoint to true under gke, although no setting is needed when the cluster has no IP-based endpoint because Cloud Deploy uses DNS by default. The VPC path keeps render and deploy inside the private network on a Cloud Build private worker pool. First create the VPC-native private cluster and set internalIp to true when the target should use the internal IP. Then create a private worker pool with network access to the cluster, such as projects/p123/locations/us-central1/workerPools/wp123. Then bind usages in delivery pipeline executionConfigs to that pool. The private pool is required for RENDER and can also serve DEPLOY and VERIFY, so adding both RENDER and DEPLOY to the same pool fixes many private-cluster blockages. Cross-project deployments add an ownership boundary where Cloud Deploy must be allowed to submit to the pool and the pool service account must be allowed to reach the cluster.

2.3Artifact Scanning Gates, Freshness, and Deploy Authentication

Artifact Analysis is the scanning service that evaluates artifacts after they arrive in Artifact Registry (opens in a new tab). Data flows from the push to Artifact Registry to Artifact Analysis, which extracts system and supported language packages and writes metadata to Container Analysis. Vulnerability knowledge is stored as a note and each instance on a specific image is stored as an occurrence, so the pipeline gate reads occurrences to allow or block an image digest. The console shows the gate in the Vulnerabilities column per digest and in Scan results with package types scanned, total vulnerabilities, fix availability, and effective severity. Results are also readable with the Google Cloud CLI or Container Analysis API. If a high-severity finding blocks promotion, patch the base image or dependency, push a new digest, wait for a fresh scan, and promote only the new digest.

Scanning depends on project APIs and repository settings. First enable the Artifact Registry API and Container Scanning API, which needs serviceusage.services.enable through Owner or Service Usage Admin. Then create a Docker repository to receive images. When the Artifact Analysis API is enabled, new images pushed to Artifact Registry repositories are scanned automatically. Language packages pushed directly stay unscanned by default until scanning is enabled on that individual repository. Enabling the API also enables language scanning for Node.js (npm) and Python packages in Docker containers in Artifact Registry, while Container Registry is not supported for those scans.

Scan metadata moves through fresh, stale, and archived states. Artifact Analysis updates metadata while the image or package was pulled within the last 30 days. After 30 days without a pull, results are stale and may block on outdated severity. After more than 90 days stale, metadata is archived and visible only through the API. An image with stale or archived metadata can be re-scanned by pulling it, with refresh taking up to 24 hours, while packages cannot be re-scanned and need a new published version. Deploy authentication depends on two controls that must both allow the request: IAM permissions that govern which identities can pull or push, and access scopes that limit OAuth methods from a VM using application default credentials. Same-project runtimes are preconfigured, while cross-project access, read-only storage identities needing upload, or custom service accounts need manual configuration. To view scans an identity needs Container Analysis Occurrences Viewer, Notes Viewer, Artifact Registry Reader, and scanner agent context, while secret scanning needs Artifact Registry Writer plus Occurrences Viewer.

2.4Cloud Build Logs, Debug State, Timeouts, and Service Identity

Cloud Build execution failures are traced through logs, live worker state, timeout settings, and the identities that run the build. Each build runs as orchestrated steps on a worker VM, and a step, worker, or IAM failure stops the pipeline. Logs written to standard output (stdout) or standard error (stderr) appear in the console and gcloud CLI, and on step failure the console links to the workdir directory in Cloud Storage for full output. Image Builder pipelines write to four channels: Cloud Build orchestrator logs for high-level steps, customization VM agent logs for stdout and stderr stored under the build ID in the _GCS_WORKDIR bucket, serial console logs for guest operating system (OS) boot failures when the agent cannot start, and guest OS journal logs for full Linux system troubleshooting. Data flows from the customization agent to the orchestrator to Cloud Storage, so missing agent output with boot failure points to serial logs while script failure points to agent and journal logs.

Interactive debugging preserves the failing worker for direct inspection. By default Image Builder deletes the worker VM immediately on failure, but setting debug: true under infrastructureConfig in imagebuilder.yaml pauses deletion. The workflow runs in order: enable the flag with machineType such as e2-standard-4, zone such as us-central1-a, and instanceDurationHours such as 1.5; submit the build and wait for the orchestrator to pause on script error until duration expiry; then connect with gcloud compute ssh WORKER_VM_NAME --zone=BUILD_ZONE --project=PROJECT_ID, adding --tunnel-through-iap when externalIP: none removes the external address. Once connected, inspect files, processes, and installer history and test fixes on disk. Without interactive access, use the automatically exported debug image image-builder-debug-image-BUILD_ID in family image-builder-debug-images, create a temporary VM from that family, connect over SSH, and isolate which customization step left the disk bad.

Timeouts and identity complete the triage. The default Cloud Build timeout is 1 hour (3600s), and a job exceeding it stops with TIMEOUT status until the root-level timeout field in cloudbuild.yaml is increased, for example to 7200s for 2 hours. Choose a larger machineType when customization is compute-bound, then raise timeout to match. The Cloud Build service account governs resource access during steps; unspecified builds may use a broad default with access to Cloud Source Repositories and Cloud Storage, while a user-specified account follows least privilege for tasks such as pushing to Artifact Registry. When a custom account runs the build, store logs in an accessible user-project bucket. The Cloud Build service agent service-PROJECT_NUMBER@gcp-sa-cloudbuild.iam.gserviceaccount.com is distinct from the execution account: the agent calls Google Cloud APIs on behalf of the build. Common signals map to fixes: storage.objects.get denial needs Project Viewer plus Cloud Build Editor or a user bucket; iam.serviceAccounts.actAs failure needs impersonation for Cloud Run or App Engine targets; cloudfunctions.functions.get 403 needs Cloud Run functions Developer; cloudbuild.builds.create denial needs Cloud Build Service Account role; deleted agent needs gcloud beta services identity create --service=cloudbuild.googleapis.com plus binding roles/cloudbuild.serviceAgent. Non-IAM blocks include Couldn't read commit for a missing branch, Pub/Sub trigger blocked by organization policy, and fatal: could not read Username on private git pull where the credential helper is removed after clone and tokens or SSH keys must be an explicit step.

Application latency and runtime failures require separating slow code, saturated infrastructure, and slow dependencies before changing scaling or code. Telemetry flows from users to Cloud Run revisions or App Engine services, then to Cloud Monitoring (opens in a new tab) for aggregate impact, Cloud Trace with Cloud Logging for per-request paths, and Cloud Profiler for function cost, while decisions flow back to scaling settings or code fixes. Adding capacity without that separation can overload a backing database or API.

3.1Scoping Latency with Monitoring, Trace, Profiler, and Layered Metrics

Cloud Monitoring quantifies whether latency affects all traffic or only a tail before deeper diagnosis. Cloud Run is integrated automatically with no setup and exposes a Request latencies dashboard with Cloud Run Revision > Request_latencies > Request latency. App Engine exposes 50th, 95th, and 99th percentile latency, so high values on all three mean at least 50% of requests are slow while high values only at 99th mean about 1% are slow. A spike must be read in its time window because a large spike in a small window can be negligible in a larger window.

Cloud Trace and correlated logs then isolate one slow request and attribute time to code, dependency, or platform. Correlated container logs tied to the request log identify the request and its trace, while the traceSampled=true filter in Logs Explorer links sampled logs to traces. Each Cloud Run request log carries an instanceId identifying the handling instance, which matters because one instance can handle concurrent requests and share global-scope state such as a single connection-pool object. Without explicit logging of outbound calls, wait time on another service appears as latency in Cloud Run itself. Triage in order: check blast radius in Monitoring against percentiles and windows; isolate one request in Trace and Logging by traceSampled=true and by instanceId under Cloud Run Revision or Job; then attribute the span to code, dependency, or cold-start and global-state reuse before scaling.

Cloud Profiler is a statistical low-overhead profiler that continuously gathers CPU and memory-allocation data from production and attributes it to source code. Flame-graph workflows for focusing, comparing across deployments, and viewing trends show whether a hotspot is new or persistent. Layered metrics complete the picture: infrastructure CPU, memory, and disk writes for overload; operator-defined app metrics such as query time collected with OpenTelemetry; service-mesh interaction metrics between microservices observed with Cloud Service Mesh; and end-to-end checks of user actions for perceived availability. When application-blocking latency is high while server latency is low, suspect client queuing, high client CPU, undersized channel pools, or intermediate network paths. Remediation matches cause: raise minimum instances to keep warm instances and avoid cold starts, limit maximum instances to reduce concurrent connections to a constrained backing service, and fix code such as reused globals, oversized pools, missing indexes, or Profiler hotspots.

3.2Container Health Signals and Lifecycle Log Correlation in GKE

Troubleshooting containerized runtime failures in Google Kubernetes Engine (GKE) means correlating live health signals from Cloud Monitoring with historical evidence from Cloud Logging. Monitoring metrics show restarts, throttling, or memory pressure, while Logging preserves kubelet, runtime, and application event sequences. Together they reveal whether code, probe configuration, or resource limits caused startup failure or crash loops.

Container health signals quantify use of configured requests and limits:

• kubernetes.io/container/restart_count tracks restarts to detect crash loops from errors, misconfiguration, or exhaustion.
• kubernetes.io/container/memory/limit_utilization tracks fraction of memory limit, capped at 1.0, for OutOfMemory (OOM) risk near 1.0.
• kubernetes.io/container/memory/used_bytes tracks actual bytes to spot leaks or growth toward the limit.
• kubernetes.io/container/memory/page_fault_count tracks major faults from disk as early pressure.
• kubernetes.io/container/cpu/limit_utilization tracks fraction of CPU limit, can exceed 1.0, for throttling.
• kubernetes.io/container/ephemeral_storage/used_bytes tracks temporary disk to prevent evictions.
• kubernetes.io/container/cpu/request_utilization and memory/request_utilization track fractions of requests, can exceed 1.0, for rightsizing and scheduling.

Log sources cover different lifecycle stages. Kubelet logs from the node service record startups, probe failures, OOM events, volume mounts, and NotReady causes. Containerd logs from the runtime record image pulls and pre-start errors. App logs from container stdout and stderr record crashes inside the process. Audit logs from the API server record who created or changed Deployments. Diagnose in order: check restart_count with memory and CPU utilization; review app logs for the failed Pod; review kubelet and containerd logs on the target node for probe, OOM, scheduling, or pull errors; then check audit and node pressure when app logs are empty. Empty app logs with a Pod that never starts points to image or scheduling pressure, not code.

3.3Runtime Exceptions, Structured Request Tracing, and IAM Integration Audits

Isolating runtime exceptions means deciding whether code or a dependency at a trust boundary failed. Cloud Error Reporting isolates code errors by grouping crashes, structured log parsing traces one request across services, and IAM audits prove whether the caller identity can still reach the dependency. Together they separate bugs from timeouts, denials, or network blocks.

Cloud Error Reporting (opens in a new tab) aggregates errors from App Engine standard and flexible, Cloud Run, Cloud Functions, Compute Engine, GKE, and languages including Go, Java, Node.js, Python, PHP, Ruby, and .NET, grouping repeated stack traces into one issue so regressions stand out. Mobile errors flow through Firebase Crashlytics instead. Data flows either from a formatted entry in Cloud Logging read automatically or from a direct Error Reporting API call. Reporting requires the Error Reporting Writer role (roles/errorreporting.writer) and cloud-platform scope. For GKE the cluster must be created with that scope and a service account holding the role with no later change, while for Compute Engine the VM must be stopped, edited to attach the Writer service account with full API access, and restarted. Without both, crashes stay invisible.

Structured tracing links the same request across Logging and Trace. Cloud Logging holds request and application entries while Cloud Trace holds timing, joined by trace, spanId, traceSampled, and request_id fields. Each Cloud Run request log contains instanceId, so filtering to one instance in Logs Explorer by selecting Cloud Run Revision or Job, expanding the entry, and choosing Show matching entries reveals whether failures follow cold starts, concurrency, or shared global state. The traceSampled=true filter then links logs to the slow trace. Service-to-service audits prove the remaining boundary: Cloud Audit Logs stores AuditLog objects in protoPayload, with Admin Activity logs always enabled for config changes, Data Access logs disabled by default except BigQuery for data reads, and System Event logs generated by Compute Engine such as live migrations. Viewing needs Logs Viewer with least privilege, retention decides history, and exports to Cloud Storage, BigQuery, or Pub/Sub extend it. A sink filtering organization SetIamPolicy calls to Pub/Sub triggering a Cloud Run function gives near-real-time notice of permission changes. Connectivity Tests checks VPC, peering, VPN, and Cloud Run to VM or managed-service paths to prove firewall blocks versus IAM denials, while Policy Troubleshooter compares principal, permission, and resource using the underlying permission rather than role name, and IAM Recommender context over 60 days explains denials after unused-permission removal.

Proactive observability turns telemetry into incidents before users notice degradation. Data flows from metric data in Cloud Monitoring or log data in Cloud Logging (opens in a new tab) into an alerting policy, then from the policy to an incident record, then from the incident to each notification channel such as email, Cloud Mobile App, PagerDuty, or Slack. Mature coverage combines threshold checks for known limits, forecast and absence checks for exhaustion and silence, expressive queries for service-level objective (SLO) burn, and synthetic probes for external availability.

4.1Alerting Policy Components, Lifecycle, and Synthetic Validation

An alerting policy is defined by its condition, notification channels, and documentation. The condition specifies data source, filters and grouping, aggregation, and static or dynamic threshold, and can monitor one metric, multiple metrics, or a ratio. Channels list destinations for action, while documentation controls subject and body where teams embed playbook and dashboard links. Policies are managed through the console, Cloud Monitoring API, CLI, or Terraform, with integrations supplying recommended policies. Monitoring evaluates continuously with no business-hours-only schedule, so teams create a snooze to suppress temporarily. If an incident is open and the condition stops being met, Monitoring moves it to closed automatically and sends closure notice. Policies can evaluate time-series metrics, individual log entries, or SQL query results over logs in Public Preview, such as latency above two seconds for five minutes creating an incident and emailing support while tolerating brief fluctuation.

Synthetic monitors test from outside before real traffic fails. An uptime check probes HTTP, HTTPS, and TCP endpoints for responsiveness, while a broken-link checker crawls pages for broken links. Because probes run independently of user load, they catch network, DNS, certificate, and routing failures that internal metrics miss. Failing probes trigger the same alerting workflow, creating incidents with latency details and notifying channels, and they are versioned with console, API, CLI, or Terraform like policies.

4.2Time-Series Conditions and Precise PromQL Alerting

Time-series conditions provide four triggers that answer different failure questions from the same store. Threshold conditions catch known bad states, absence conditions catch silence, PromQL conditions catch complex ratios, and forecasted conditions buy lead time before exhaustion.

A PromQL condition in Cloud Monitoring evaluates any valid expression, including combinations and dynamic thresholds based on another metric. Prometheus metrics reach Monitoring through Google Cloud Managed Service for Prometheus or through the Ops Agent with Prometheus or OpenTelemetry Protocol (OTLP) receivers. Precise queries depend on label choice: if a resource has a cluster label and the metric has kubernetes_cluster_name, filter on the resource cluster label for consistent series. Do not use system- or user-defined resource metadata labels for sensitive policies because they populate asynchronously and can make triggering erratic.

4.3Structured Logs, Custom Metrics, and Distributed Traces

Optimizing observability data means shaping logs, metrics, and traces so failures are found quickly without storing unnecessary telemetry. In-process instrumentation generates telemetry inside the application and exports it to a Google Cloud project, where Cloud Logging stores logs, Cloud Monitoring stores metrics, and Cloud Trace stores traces. Most services generate basic signals automatically, so optimization adds detail where it speeds diagnosis and aggregates elsewhere.

A structured log entry in Cloud Logging is a timestamped record combining message with severity, customer ID, error message, stack trace, and code location. Severity and labels let operators filter high-impact failures, while Error Reporting groups entries into error groups with cause and history. Logs Explorer views individual and related entries and annotates error-group and trace links, while Observability Analytics queries log data with SQL for trends such as average HTTP latency by URL. Two notification paths turn logs into action: a log-based alerting policy fires on a single rare entry such as service-account key access or deployment, while a log-based metric counts entries or extracts histograms and an alerting policy monitors the trend such as error counts. Custom metrics are application-defined numerics such as latency, queue depth, token usage, or success rate measured regularly. Instrumentation code generates telemetry and exports it, with OpenTelemetry as the vendor-neutral standard across languages. Google Cloud Managed Service for Prometheus handles third-party metrics for GKE such as Postgres or Redis exporters, the Ops Agent collects Prometheus and OTLP plus host CPU, GPU, memory, and process metrics on Compute Engine, and Cloud Run uses Prometheus or OpenTelemetry sidecars. Cover the four golden signals of latency, traffic, error rate, and saturation when defining customs.

A trace in Cloud Trace is the path of one end-to-end request composed of spans where each span records one function or operation. Following spans in order shows service-to-service flow and where latency accumulates, including agent actions such as Model Context Protocol (MCP) calls with derived model-call and token counts. Instrumentation exports spans to the project where Cloud Trace visualizes per-span latency. When spans point to slow blocks, Cloud Profiler analyzes CPU and memory to guide optimization, and correlated structured logs with the same request context reveal the exact error and code location.

4.4Correlating Anomalies, Handling Missing Telemetry, and Controlling Cost

Diagnosing root causes from anomalies correlates metrics, logs, and traces to separate infrastructure from application failures. Correlation starts with shared identity: Cloud Logging access entries carry trace as projects/YOUR-PROJECT/traces/TRACE-ID, spanId, traceSampled, request_id, and logName, Cloud Trace stores the same trace ID, and Cloud Monitoring shows the spike for the same window. From Logs Explorer select View trace details, or from Trace explorer open the span Logs and Events tab. Server access logs under k8s_container as server-accesslog-stackdriver give the server-side view, while client access logs under k8s_pod as client-accesslog-stackdriver give the client-side view, both with URL, size, latency, headers, workload identity, and response_flag. Comparing views separates mesh or network problems from handling problems.

Latency diagnosis for APIs moves from aggregate to conversation in order: open Enabled APIs and Services Metrics and view Traffic by API method, enable Admin Read, Data Read, and Data Write audit logging so calls emit detail, then drill into high-latency conversations with Log Analytics filtering TIMESTAMP_DIFF such as over 10 seconds and Logs Explorer filtering protoPayload.methodName and Timestamp. Without Data Read and Write entries there are no per-conversation records. Control-plane versus data-plane separation uses istiod metrics prefixed istio.io/control and pilot_proxy_convergence_time for delay between config change and proxy convergence; rising convergence with disconnects points to control plane, while flat control plane with rising data-plane latency points to application. Check via the sample Istio Control Plane Dashboard and by querying http://localhost:15014/metrics in the pilot pod filtered for convergence.

Missing logs are themselves an anomaly. Configuration causes include disabled Cloud Logging API, disabled cluster logging, node pool lacking scope, or service-account 401 or 403 in agent logs. Resource causes include RESOURCE_EXHAUSTED write quota or node exceeding agent throughput, CPU, or memory. Customer-side filtering causes include exclusion filters dropping matches, fully buffered stdout from piping delaying logs until exit, overly restrictive Logs Explorer filters, and code not writing to stdout or stderr. Platform causes include retention expiry, Cloud Logging service delay, or unsupported GKE version, with gcpdiag inspecting for misconfigurations. Log entries can appear minutes before matching traces, so retry trace lookup before concluding absence. Control volume and cost with organization log sinks for compliance, BigQuery exports for fast analysis, Observability Analytics SQL over buckets, Data Access audit logs for sensitive access, and Fluent Bit or ingestion exclusions to strip Social Security numbers, card numbers, and emails while exporting unfiltered copies when required. The Metrics Management page reports billable ingestion by bytes and samples, cardinality, reads, alert and dashboard use, and write errors to guide exclusions, while Managed Service for Prometheus charges per sample ingested and Fluent Bit or BindPlane logs charge at Logging rates with four-hour local buffering and retry.

Performance work in Google Cloud answers requests as close to the user as possible and matches compute and database capacity to real demand. Traffic flows from users to the global front end and Cloud CDN (opens in a new tab) cache check, then only on cache miss to backends, while layered caches and tuned machine types absorb repeated reads inside the region. Together edge termination, caching, and rightsizing shorten round trips and protect origins during spikes.

5.1Latency Segments, Edge Caching, and Global Deployment Choices

Latency segments show where caching helps and where it cannot. Last-mile latency from user to provider depends on access technology with typical 1-10 ms on modern networks, 10-60 ms on DSL and cable, and 100-500 ms on 3G cellular. Google frontend and edge point of presence (PoP) latency covers the path to Google's edge where global load balancing terminates TCP and SSL and Cloud CDN serves hits. Compute Engine region selection covers only edge-to-region transit inside Google's network plus app processing. Region moves cannot fix total latency when the last mile dominates, while edge caching is powerful because many round trips end at the PoP and skip edge-to-region and app segments.

Cloud CDN is the edge-caching service that works with the global external Application Load Balancer to accelerate HTTP(S) from edge PoPs. A request flows from user to global front end, through Google Cloud Armor edge security, to the Cloud CDN cache check. On hit, content returns without contacting origin; on miss, backend security evaluates and the backend fulfills per policy. The global external Application Load Balancer is a proxy Layer 7 balancer with URL mapping and SSL offloading where the session terminates at Google Front Ends (GFEs) at the edge and traffic proxies to backends. Configurable backends include two MIGs in different regions plus GKE, Cloud Run, Cloud Run functions, App Engine, on-premises, and other clouds with traffic splitting across versions.

Global load balancing requires Premium Tier in Network Service Tiers because Standard Tier enters Google's network near the destination region rather than near the user. Behind the edge, layered caching absorbs repeated reads: edge caching before the application, Memcache before slower persistence such as Datastore, and in-instance memory inside the instance. A Memcache flush that pushes reads to Datastore spikes latency at the dependency, so monitor dependencies and shed load with adaptive throttling when a non-scaling SQL database risks overload. Deployment patterns trade latency against replication: single-region frontend and backend with CDN remains best in many cases with a second region only for backup; distributed frontend with single backend works when requests need little backend data via frontend caching or async writes; fully distributed frontend and backend minimizes latency but needs full local replication such as Spanner three-continent multi-regional with read-write replicas in the US and read replicas in Europe and Asia; multiple parallel apps preserve latency with little sharing via country domains, game rooms from a global lobby, or region-specific subdomains.

5.2Defining Signals and Isolating Application, Network, and Dependency Bottlenecks

Diagnosing latency separates total delay into application, network, and external-service bottlenecks using Cloud Trace spans, VPC Flow Logs round-trip time, and Cloud Monitoring metrics. A request typically flows from client through internet to GFE to serving infrastructure to service instance to backing dependencies, so compare each layer before changing code or capacity. Server latency measures work inside the service from receipt to last response byte. Operation latency measures total client-to-service-to-client round trip including retries, client-library, network, and server time. Workload patterns can raise operation latency while infrastructure stays healthy, such as a scan growing from a hundred rows to a million after import. High operation latency with flat server latency points to client, network, or dependencies, while slow client consumption of large responses can inflate server latency. Always compare the same percentile and window.

Scope the incident by application, service, version, endpoint, client, and window using 50th, 95th, and 99th percentiles, where high values on all three mean at least half of requests are slow and high values only at 99th mean about 1% are slow. Verify the application sits in the same zone as its database cluster to minimize distance, then compare in order: check server read and write latency in Monitoring sliced by cluster with requests, throughput, and node count; check client-side operation latency such as bigtable.googleapis.com/client in Metrics Explorer or Cloud Run Revision Request latencies; then correlate logs and traces for the same requests with traceSampled=true and instanceId filtering when concurrency shares instances. If monitoring shows latency but request logs do not, suspect upstream load balancing or severe instance failure that stopped routing and inspect logs just before the incident. Cloud Trace breaks one request into frontend, code, and downstream spans, so add explicit logging around calls to databases and APIs or external delay looks local. In App Engine the path flows Client to Internet to GFE to serving infrastructure to Service instance, where errors with flat instance scaling mean overload. In Cloud Run shared globals such as one connection pool bind latency to one instance, revealed by instanceId filtering for cold starts and queuing.

Network measurement separates transit from processing. VPC Flow Logs measures TCP round-trip time between connection ends, while HTTP(S) load balancing exposes frontend round-trip between load balancer and client and backend latency to backends, plus user headers for country and estimated round trip. Premium Tier uses Google's private global network with lower latency and is required for global balancing, while Standard Tier uses transit ISPs. Analyze paths with traceroute, MTR, or TCP traceroute per hop, curl timing to the nearest GFE, tests from different geographies, and simultaneous tcpdump or Wireshark captures including a PCAP sidecar in the same network namespace as Cloud Run without competing for primary resources. Connectivity Tests checks Cloud Run to VM or managed-service paths for misconfiguration. Geography matters: hostnames resolve to the closest GFE by client IP without EDNS0, HTTP/2 parallel requests can hit head-of-line blocking at the GFE needing QUIC, custom domains add an SSL hop at the GFE versus appspot.com, and planning estimates about 1 ms round trip per 100 km times 1.5 to 2 for routing, with Europe-to-Asia often via the US. Same-zone c2-standard-4 VMs typically measure under 55 microseconds round trip at 50th percentile and under 80 at 99th, dropping to under 45 and under 60 with compact placement.

Client queuing raises operation latency with flat server latency. In Cloud Bigtable check Application Blocking Latencies; high values matching the report focus on the client, with GKE or Compute Engine clients escalated after client causes are ruled out. Tune gRPC channel pools iteratively because too few queue buffered requests while too many raise attempt latencies, enable channel priming on Java client 2.17.1 or earlier with auto-refresh from 2.18.0, and on 2.21.0 later client_blocking_latencies includes gRPC queue time. If connectivity_error_count is high the app struggles to reach the frontend, so lower RPC timeouts to retry on different channels without dropping far below normal P99. If retry_count is high check attempt_latencies status where DEADLINE_EXCEEDED means deadlines shorter than average attempt latency. If no metric exceeds norms, suspect small pools or high client CPU queuing on gRPC threads. Serverless adds cold-start delay: raise Cloud Run minimum instances to stay warm, limit maximum to protect backing services, and fix slow handlers via Profiler, while App Engine overload comes from traffic outpacing autoscaling, batch spikes, CPU-scaling of I/O-bound flexible services, or migrating before caches warm without reusing version names that restart all instances together.

5.3Rightsizing Compute, Autoscaling, and Cloud SQL for Sustained Throughput

Optimizing resources matches compute, scaling, and database capacity to demand so latency stays low without overspending. Too little creates CPU, memory, or I/O saturation, while too much wastes budget. Right-size Compute Engine by choosing machine types to CPU and memory profiles using utilization recommendations, with custom machine types provisioning cores and memory independently to avoid paying for unneeded resources. Independence also limits license cost by limiting cores and by disabling simultaneous multithreading (SMT) to reduce vCPUs per core from 2 to 1. Disk throughput and IOPS scale with disk size and vCPU count, so an undersized target causes long migrations and database errors even when CPU looks healthy. When a well-sized VM still saturates, scale out with groups rather than growing one VM.

Plan capacity before autoscaling can react. For a MIG deployment decide MIG size, VM machine type, and persistent disk number, type, and size with cost via the Google Cloud Pricing Calculator. Forecast from historical trends and new-geography growth, seasonal variations, and event spikes under normal versus peak conditions. Include N+1 redundancy with one extra component beyond minimum at every layer, per-project quotas as resource quotas for creatable maximums and rate quotas for API requests per period at zonal, regional, or global scope, and reservations for assured zonal capacity for a chosen machine type that can be project-specific or shared. Benchmark with load tests under normal and peak because efficiency relating load to CPU and memory degrades at high load and drifts over time.

Autoscaling of stateless MIGs adds VMs when traffic rises and removes them when demand falls, while stateful MIGs preserve per-VM state and cannot be autoscaled and must be sized manually. Spot VMs match regular performance at lower cost but can be preempted, and MIGs may fail to reach target size when zonal capacity is limited. GKE separates layers that depend in order: HorizontalPodAutoscaler (HPA) first increases Pods on CPU or custom metrics such as 70% or 80% to leave buffer, then Cluster Autoscaler adds nodes only if Pods stay unschedulable, then scale-down consolidates and removes underutilized nodes. VerticalPodAutoscaler (VPA) resizes container requests for non-horizontal workloads, and node auto-creation creates pools shaped to pending Pods to reduce stranded capacity.

Safe operation needs bounds: run VPA in Off recommendation-only mode at least 24 hours and ideally one week before Initial or Auto with minimum and maximum, and use PodDisruptionBudget (PDB) plus minimum node count so consolidation preserves night capacity. Cloud SQL tuning adds sizing, locality, query, and storage controls. Performance-intensive workloads need at least 60 GB memory to avoid pressure, distance slows writes between writer and database and reads even more between reader and database, and very large table counts degrade performance and risk SLA loss. Query Insights in Cloud Monitoring with engine tools identifies expensive commands, autogrow in megabytes with matched increments prevents unchecked file growth versus percentage growth, SSD is the efficient default while HDD fits only 10 TB or larger latency-insensitive or infrequent datasets, Cloud SQL Enterprise Plus edition raises machine limits and adds data cache for higher throughput, and automatic storage increases with peak buffer prevent outages. Detect constraints with Cloud Monitoring dashboards including Cloud SQL monitoring plus Active Assist recommendations, Cloud Profiler for function-block hotspots when infrastructure looks normal, Cloud Trace for microservice flow, and Cloud Logging for event context, fixing queries or functions when Profiler points to code and adjusting machine type, replicas, policy, or edition only after confirming true CPU, memory, or I/O constraint.

Key Facts

• Regional persistent disks replicate to two zones in a region while Filestore Enterprise tier is regional and Basic and Zonal tiers are zonal without cross-zone replication.
• Google Cloud quotas are per-project zonal, regional, or global limits that block scale-out while Compute Engine reservations provide assured zonal capacity for a chosen machine type.
• Cloud Deploy calls skaffold render to produce manifests and skaffold apply to deploy them, and private GKE deployment requires a private worker pool for RENDER with DEPLOY optionally on the same pool.
• Scan metadata becomes stale after 30 days without a pull and archived after more than 90 days stale, where only images can be re-scanned by pulling and archived results need the API.
• High operation latency with flat server latency points to client, network, or dependency delay, confirmed with VPC Flow Logs round-trip time and Cloud Trace dependency spans.