Troubleshooting Failed Cloud Deployments

Troubleshooting Infrastructure as Code and pipeline execution failures means restoring a deployment to a known-good state and finding which layer failed: configuration, build stage, identity, quota, or network. In Google Cloud, Infrastructure Manager applies Terraform configurations as versioned deployments, while Cloud Build orchestrates pipeline steps and Image Builder (opens in a new tab) runs image-customization work on a temporary worker virtual machine (VM). Systematic diagnosis moves from deployment prerequisites to stage logs to interactive debugging, then to authentication, quota, and connectivity checks.

1.1Update and Restore Infrastructure Manager Deployments

An Infrastructure Manager deployment update reapplies a Terraform configuration to correct drift or to reach a new desired state. Drift occurs when resources change outside Terraform, for example when someone edits a resource in the console, and reapplying the original configuration returns the infrastructure to the chosen state. Applying a new or edited configuration adds resources or changes settings to reach a new desired state.

A failed update is often a missing prerequisite rather than a syntax error. The calling identity needs the roles/config.admin role, and the service account that calls Infrastructure Manager needs permission to use the Infrastructure Manager service plus permission for the Google Cloud resources defined in the revision. When the revision defines different resources than the current deployment, the service account needs permissions for resources in both configurations. Other common blockers include:

• Infrastructure Manager API is not enabled for the project.
• Terraform configuration violates constraints, including the rule that the configuration must not contain sensitive data.
• Organization quota for a product used by the intended infrastructure is exceeded.
• Data migration or service continuity steps for modified resources were skipped.
• Terraform configuration is stored in a private Git repository that is not connected to Cloud Build.

Preview an update before applying it when the change modifies production resources. Preview shows the planned effect without configuring resources, so quota, permission, and dependency problems can be corrected before the revision runs.

1.2Diagnose Pipeline Stages With Logging Channels

Pipeline stage diagnosis starts by matching the failure symptom to the logging channel that records that layer of the stack. Image Builder writes to multiple locations because a high-level orchestrator failure, a script error, a boot failure, and an operating-system failure each leave evidence in a different place. Traffic flows from the customization agent on the worker VM to Cloud Build trace logs and to a Cloud Storage workdir bucket, and the failing step prints a link to its workdir directory.

When orchestrator logs point to a workdir directory, open that build ID folder before changing code. Agent output answers whether a script command failed, serial output answers whether the VM booted, and journal output answers whether a system service or kernel condition contributed to the failure.

1.3Interactively Debug Script Failures and Timeouts

Interactive debugging keeps a failed worker VM alive so its live filesystem and running processes can be inspected. By default, if a customization step fails, Image Builder terminates execution and deletes the worker VM immediately to control costs. Setting debug: true under the infrastructureConfig block in imagebuilder.yaml changes that lifecycle from failed then deleted to failed then paused.

The ordered debug workflow is:

1. Enable the debug flag: Set debug: true under infrastructureConfig, with supporting fields such as machineType, zone, and instanceDurationHours to bound how long the VM remains running.
2. Trigger the build and wait for failure: Submit the build normally; when a script error occurs, the orchestrator pauses execution and leaves the worker VM running in the project instead of deleting it.
3. Connect to the worker VM to debug: Use SSH to connect directly to the active guest system, reproduce the failed command, inspect files, and identify compile-time or script errors.

After interactive inspection, analyze intermediate debug images when the failure leaves a partially customized disk that can be compared against a working image. Resolve Cloud Build timeouts separately from script logic by checking whether the step needs more time, a larger worker, or a split into smaller steps, because a timeout can mask an otherwise correct script.

1.4Resolve Authentication, Quota, and Connectivity Blockers

Authentication, quota, and connectivity blockers prevent Infrastructure as Code and pipeline stages from starting or reaching Google Cloud APIs even when the configuration is correct. Authentication verifies identity for access to services and APIs, typically through the Google Cloud CLI with gcloud init and a default region and zone, plus the service account used by Cloud Build and Infrastructure Manager. Required Google services must also be enabled, because a deployment tool can succeed locally while the target project still blocks the call.

For service-dependent pipelines, confirm that servicemanagement.googleapis.com (Service Management API) and servicecontrol.googleapis.com (Service Control API) are enabled with gcloud services list, then enable them with gcloud services enable when missing. The instance or worker running the proxy or builder also needs permission to call Service Management and Service Control, so a 401 or 403 error points to the node or build service account rather than to application code. When a Terraform configuration is pulled from a private host, the Git host and repository must be connected to Cloud Build, otherwise the revision fails before any resource is planned.

Quota and network policy form the next boundary. The project can exceed Cloud Logging API write quota with RESOURCE_EXHAUSTED errors, exceed product quota checked before an Infrastructure Manager update, or exceed node throughput and CPU or memory needed by logging and build agents. Virtual Private Cloud (VPC) firewall rules allow or deny traffic per instance across the VPC network, network tags select which VMs receive firewall rules and routes, and VPC Service Controls creates a service perimeter around services such as Cloud Storage and BigQuery that controls what is allowed in and out. Narrow the scope first by asking whether the failure is project-wide or isolated to one cluster, node pool, or pipeline target, whether it began after a recent configuration change, and whether it correlates with a traffic spike or high CPU or memory use.

Rollback and canary analysis strategies in Cloud Deploy (opens in a new tab) limit user impact by detecting a bad release on a small subset of traffic and stopping its progression before full rollout. In Cloud Deploy, a release moves through a rollout that contains phases, jobs, and job runs, and each level reports its own lifecycle state for troubleshooting. Canary analysis uses those states plus verification and mitigation operations to decide whether to advance, halt, or recover the deployment.

2.1Canary Analysis to Contain Faulty Releases

Canary analysis in Cloud Deploy is a deployment strategy that splits traffic between the existing stable version and a new canary version so metrics and errors can be compared before full rollout. Traffic flows from the load balancer or service mesh to both versions during the canary phase, then flows only to the new version after the rollout advances to the stable phase. For targets configured with a strategy other than standard, an operator must advance the rollout from phase to phase, which creates an explicit decision point to analyze canary metrics before promotion.

Cloud Deploy changes canary behavior when there is no running version to split traffic with. In that first-deployment case, Cloud Deploy skips the canary phases and proceeds directly to the stable phase. The skipped phases and their jobs transition to SKIPPED, which means no canary comparison occurred rather than a failure. This distinction matters for troubleshooting because a SKIPPED canary does not indicate a performance regression.

2.2Rollout and Phase States for Troubleshooting

A Cloud Deploy rollout tracks the overall delivery of a release to a target, while a phase tracks one stage within that rollout. The rollout depends on its phases, and a phase depends on its jobs, so a failure at a lower level propagates upward and determines the mitigation path.

A rollout reports one of the following states:

• PENDING: The rollout has not begun processing and will transition to IN_PROGRESS or CANCELED.
• PENDING_APPROVAL: The rollout requires approval but has not yet been approved.
• PENDING_RELEASE: The rollout is waiting for the release to be rendered.
• IN_PROGRESS: A job run is processing.
• FAILED: A job failed and the failure was not ignored.
• SUCCEEDED: The rollout has finished with no failures.

A phase reports one of the following states:

• PENDING: The phase is waiting for another phase in the rollout to finish.
• IN_PROGRESS: The phase has started.
• SUCCEEDED: The phase completed successfully.
• FAILED: A job in the phase failed and the failure was not ignored.
• ABORTED: A previous phase failed, so this phase will not run.
• SKIPPED: A canary phase was bypassed because no running version existed for traffic splitting, and execution moved to the stable phase.

When a phase enters FAILED, subsequent phases transition to ABORTED. When a phase enters PENDING_APPROVAL or PENDING_RELEASE, the rollout waits and no further phases start until the approval is granted or the release render completes.

2.3Job and Job Run States for Failure Diagnosis

A Cloud Deploy job tracks a single unit of work inside a phase, such as deploy or verify, while a job run tracks one execution attempt of that job. The job depends on its job runs, so the job state summarizes whether the work succeeded, failed, was bypassed, or was explicitly excused.

A job reports one of the following states:

• PENDING: The job is waiting to begin because another phase or job has not finished.
• IN_PROGRESS: A job run for this job is currently running.
• SUCCEEDED: The job run finished successfully and the next job, phase, or rollout completion can proceed.
• FAILED: A job run failed and the failure was not ignored, or the user terminated the job run.
• IGNORED: A job run failed but the user chose to ignore the failure, allowing the rollout to continue.
• ABORTED: A prior job or phase failed without an ignore, so this job will not run. For example, if a phase includes a deploy job and a verify job and the deploy job fails, the verify job is aborted.
• DISABLED: The job is present but not active. Phases always include verify jobs, and the verify job is set to DISABLED when verification is not enabled.
• SKIPPED: The job sits inside a canary phase that was bypassed for lack of a running version, so execution moved to the stable phase.

A job run reports one of the following states:

• IN_PROGRESS: The job run has begun but has not finished.
• SUCCEEDED: The job run finished successfully without failing or being terminated.
• FAILED: The job run failed during execution.
• TERMINATING: The user requested termination but termination has not finished yet.
• TERMINATED: The user terminated the job run.

2.4Mitigation Operations to Restore a Stable State

Mitigation operations in Cloud Deploy stop a faulty rollout from progressing and return the delivery pipeline to a known good condition. The operations act on different levels of the hierarchy: the rollout control plane manages phase progression, while job-level actions manage individual failures. Order of operations matters because a terminated or failed job blocks downstream jobs and phases until it is retried, ignored, or the rollout is canceled.

The available operations are:

• Advance the rollout: Move a non-standard rollout, such as a canary, from its current phase to the next phase after analysis shows no regression.
• Cancel the rollout: Stop the entire rollout so no further phases or jobs start.
• Terminate a job run: Stop a currently running job run, which moves the job run through TERMINATING to TERMINATED.
• Ignore a job: Mark a FAILED job as IGNORED so subsequent jobs and phases can proceed despite the failure.
• Retry a failed job: Re-execute a job whose job run entered FAILED to attempt recovery without creating a new release.

Choose advance when canary metrics show no errors or latency regression and the new version is safe to receive more traffic. Choose terminate or cancel when metrics show regressions, error spikes, or a misconfigured deployment that must not progress to stable. Choose retry when the failure was transient, such as a temporary infrastructure error, and choose ignore only when the failed check is known to be non-blocking and downstream verification will still protect the stable version.

Diagnosing deployment failures with Google Cloud observability means correlating three signals to find root cause: metrics show what changed in performance and resources, logs show what happened in APIs and applications, and traces show where a request slowed or failed across services. Cloud Monitoring collects and alerts on time-series metrics, while Cloud Logging ingests, stores, routes, and queries log entries from infrastructure, audit, and application sources. A systematic diagnosis starts with centralized collection, then filters by time and deployment version to separate infrastructure provisioning errors, application runtime issues, and configuration mismatches.

3.1Centralized Collection and Routing for Diagnosis

Centralized observability for deployments aggregates signals from projects, folders, and services into one place where failures can be compared. Cloud Logging uses aggregated sinks to collate and route log entries into a single central repository for operations teams, and intercepting aggregated sinks avoid storing high-volume logs like Data Access logs in more than one place. Log entries flow from Google Cloud services and workloads to log buckets, then from sinks to destinations such as BigQuery for analysis, Cloud Storage for long-term retention, Pub/Sub for integration with other systems, and a SIEM such as Security Command Center. Central analysis depends on audit logging being enabled, including for Cloud Storage and Compute Engine, and on VPC Flow Logs, firewall rules logging, and load balancer logging being enabled so network denials are visible alongside API and application events.

Diagnosis also depends on consistent deployment practices. Deployment scripts and orchestration tools must allocate resources as intended, with resource scheduling managed through services such as Cloud Build and Google Kubernetes Engine (GKE). When a rollout fails, the investigation checks the pipeline definition, the infrastructure actually created, and the application behavior in that order, so a missing quota, image, or permission is not mistaken for a code bug.

3.2Interpreting Cloud Logging Signals

Cloud Logging is the record of API calls, configuration changes, and application output, queried in Logs Explorer with filters for severity, resource, timestamp, and deployment label. Admin Activity audit logs record configuration changes, System Event audit logs record Google Cloud system actions, and Data Access audit logs with DATA_READ, DATA_WRITE, and ADMIN_READ record each read, write, and query event for sensitive data. Application logs must include timestamps and structured exception data, and logging filters identify successful and unsuccessful events and failed data access events. Custom logs can be created in application code with Cloud Logging libraries, and usage logs for Cloud Storage buckets capture storage access that audit logs alone may not show.

Effective log diagnosis follows an ordered audit lifecycle:

1. Enable logging: Enable audit logging for services in use, including Cloud Storage, and define retention for log buckets.
2. Create log-based counter metrics: Build counter metrics in Logging that convert matching log entries into time-series data, such as counts of Permission Denied IAM errors or KMS key revocations.
3. Configure alerting policies: Build metric-based alerting policies in Monitoring that open an incident when the rate exceeds a threshold, and deploy log-based alerting policies in Logs Explorer or Monitoring with exact filters for high-severity events such as unauthorized IAM policy modifications.
4. Create notification channels: Route incidents to notification channels that include deep-links to the Logs Explorer query, standard operating procedures, and remediation steps.
5. Analyze and export: Regularly analyze logs for anomalies, export logs through Pub/Sub or to BigQuery for deeper correlation, and review and update alerting policies after each deployment.

Log-based alerting policies also monitor administrative calls to the Cloud Logging API itself and alert on unexpected log export sink modifications or bucket deletions, which can hide a deployment failure. For application exceptions, Error Reporting SDKs integrated into code or structured JSON exception parsing in Logging group repeated stack traces, and Error Reporting notification channels or Monitoring alerting policies notify teams when a new exception class appears after a release.

3.3Interpreting Cloud Monitoring Signals

Cloud Monitoring turns metrics into dashboards, alerts, and service health views that show when a deployment degraded resources or latency. The Cloud Monitoring dashboard is used to review virtual machine (VM) configuration and to monitor demand and performance, while GKE system metrics show resource allocations, node pressure, and scheduling constraints for containers. Log-based counter metrics flow from Logging to Monitoring, where a metric-based alert starts an incident when the rate exceeds a specific threshold, for example greater than 50 failed login attempts in five minutes. Firewall insights, allowed and denied ports, and VPC Flow Logs provide the network side of the same incident, showing whether traffic was blocked before it reached the new revision.

Alert quality determines how fast root cause is found. Policies should target critical indicators of compromise or misconfiguration, such as Permission Denied IAM errors, unexpected API key creations, or sudden firewall configuration changes, rather than alerting on every informational event. Each alert depends on a notification channel with clear ownership, and channels should carry the exact Logs Explorer query, the affected project and revision, and the expected containment action so the on-call engineer moves directly from metric spike to relevant logs and traces.

3.4Distinguishing Infrastructure, Runtime, and Configuration Failures

Failure type is determined by which signal changes first and which layer records the denial or exception. Infrastructure provisioning errors originate in resource creation and scheduling, application runtime issues originate in code execution after serving starts, and configuration mismatches originate in policies, permissions, or network controls that block otherwise healthy code.

Traffic that fails before reaching the workload points to configuration or infrastructure, while traffic that reaches the workload and then throws exceptions points to runtime code or its immediate dependencies. When inbound HTTPS fails inspection or egress is blocked, firewall rules logging on threat prevention profiles routed to operations analysis helps correlate blocked network signatures with host-level telemetry to identify the targeted workload for patching or rollback.

3.5Preserving Evidence with Retention and Access Control

Evidence preservation for deployments depends on retention periods and tamper protection for log buckets. Admin Activity and System Event audit logs are retained for 400 days, while Data Access audit logs are retained for 30 days by default, so critical read and query evidence must be moved to another storage location such as BigQuery or Cloud Storage before expiration. Bucket Lock and retention policies on the custom Logging bucket prevent attackers or unauthorized administrators from deleting logs to cover tracks, and uniform bucket-level access on the bucket that stores audit logs enforces consistent access control.

Access to diagnostic data is itself controlled. IAM allow policies control who can see error messages and who can access logs and log configurations, and access to audit management information is restricted to privileged users. Without these controls, sensitive log data leaks through overly broad access, and without the retention move, post-incident review loses the Data Access history needed to prove whether a deployment read or modified the correct data.

Key Facts

• Infrastructure Manager deployment updates require the roles/config.admin role, enabled APIs, and service-account permissions for resources in both the current and new revisions.
• A failed Image Builder worker VM is deleted by default and remains running for SSH inspection only when debug: true is set under infrastructureConfig.
• Cloud Deploy marks bypassed first-deployment canary phases as SKIPPED, marks phases blocked by prior failures as ABORTED, and allows failed jobs to be marked IGNORED to continue the rollout.
• Permission Denied IAM errors in audit logs indicate provisioning or configuration denial, while a new Error Reporting exception class after a release indicates application runtime failure.
• Admin Activity and System Event audit logs are retained for 400 days, while Data Access audit logs are retained for 30 days by default and must be exported to BigQuery or Cloud Storage for longer preservation.