Auditing Deployments and Managing Pipeline Rollouts

Auditing and tracking deployments is the system that proves where an artifact came from, how it was built, where it was released, and who approved the change. Artifact Registry (opens in a new tab) stores container images and records their lineage, Cloud Build creates verifiable build history, Cloud Deploy (opens in a new tab) creates ordered release history across environments, and Cloud Audit Logs records who invoked each action. Together they provide operational visibility, security enforcement, and compliance evidence for software supply chain security (opens in a new tab).

1.1Artifact Provenance and Vulnerability Scanning in Artifact Registry

Artifact provenance and vulnerability scanning in Artifact Registry proves image origin and safety before promotion. Artifact Analysis extracts package inventory when an image is pushed and records vulnerabilities as metadata tied to the image digest, while Artifact guard evaluates images against centralized policies during build, registry storage, and runtime. Binary Authorization (opens in a new tab) then enforces the decision by blocking non-compliant images from deploying to Google Kubernetes Engine (GKE).

Enabling scanning requires both project-level and repository-level action. An identity needs serviceusage.services.enable through Owner (roles/owner) or Service Usage Admin (roles/serviceusage.serviceUsageAdmin) to enable the Artifact Registry and Container Scanning APIs. When Artifact Analysis is enabled, new images pushed to repositories in the project begin scanning, but scanning stays disabled until it is enabled for the individual repository. Viewing results requires Artifact Registry Reader for repository access, Container Analysis Occurrences Viewer for findings, Container Analysis Notes Viewer for vulnerability definitions, and the Container Scanner Service Agent for the scanning service.

Scan metadata follows an active-to-stale-to-archived lifecycle driven by pulls. The service refreshes scans while the image has been pulled within the last 30 days. After 30 days without a pull, results become stale and stop updating. After 90 more days stale, metadata becomes archived and can be evaluated only through the API. Pulling a stale or archived image triggers a re-scan that refreshes within up to 24 hours. To inspect findings, open the Repositories page, click the image name to see per-digest totals in the Vulnerabilities column, then click the digest link to see detailed occurrences and CVE details.

1.2Promotion Policy and Build Gating with Artifact Guard and Cloud Build

Artifact guard centralizes vulnerability policy and enforces it at three different scopes. A policy can block artifacts above a severity such as HIGH or CRITICAL, explicitly allow or deny packages, versions, or CVEs, grant grace periods for specific CVEs, and centralize findings from pipelines, registries, and clusters in Security Command Center. A Software Bill of Materials (SBOM), a machine-readable inventory of components, supplies the lineage that scanners use to identify CVEs. Security Administrators define policies, DevOps or Platform teams wire CI/CD connectors and GKE admission controllers, and Developers fix dependencies after build failures.

On-Demand Scanning in Cloud Build gates promotion before registry lineage exists. Cloud Build builds the image from source, invokes a scan of the built image, compares severity against the threshold, and only then flows the image from the build step to Artifact Registry storage. Because the check runs before the push, vulnerable artifacts never gain downstream lineage that Binary Authorization would later need to block.

1.3Build History, Delivery Metrics, and Approval Gates in Cloud Build and Cloud Deploy

Cloud Build is the service that runs builds on Google Cloud infrastructure in isolated, ephemeral environments controlled by IAM and VPC Service Controls. Each build emits step logs, duration, status, and toolchain details, and container builds can generate authenticated, non-falsifiable provenance at Supply-chain Levels for Software Artifacts (SLSA) Level 3 that binds the output digest to source locations, build steps, toolchain, and duration. Security insights consolidate SLSA level, artifact vulnerabilities, and provenance, so step-log failures can be correlated with source, dependency, or toolchain changes. Artifact Analysis continues to update metadata when new vulnerability data arrives, so a passing build can later show new findings.

Cloud Deploy is the continuous delivery service that automates releases to a defined sequence of targets for GKE, GKE Enterprise, and Cloud Run. A release flows from the pipeline to the first target and advances only after success and any required approval, which creates deployment history ordered by environment. Delivery metrics track release frequency, promotion success and failure by target, rollout duration, and rollback rate. Approval gates block promotion until an authorized one-click approval is recorded, so the audit trail shows who approved which release for which environment. Binary Authorization extends this to deploy time by checking attestations, which are signatures proving builder identity, vulnerability-policy compliance, or promotion criteria; when attestations are missing, GKE and Cloud Run deny deployment even if the pipeline was compromised.

1.4Audit Log Types, Access Control, and Retention in Cloud Audit Logs

Cloud Audit Logs is the Google Cloud auditing system that records who did what, where, and when with immutable entries. All Google Cloud services will eventually provide audit logs, so build, deploy, and artifact actions flow through the same framework. Google Workspace and Cloud Identity produce Admin Activity logs separately, including logins, which can be shared into Google Cloud for central management.

Access is split by sensitivity. Logs Viewer (roles/logging.viewer) grants Admin Activity, Policy Denied, and System Event logs, while Private Logs Viewer (roles/logging.privateLogViewer) adds Data Access logs in the _Default bucket. Retrieval follows an ordered flow: open Logs Explorer, select the resource type such as k8s_cluster for Binary Authorization pod decisions, select the log name, then refine the query and time range. For enforcement diagnosis, rejected pod creation is found by filtering protoPayload.methodName for pod create or update with Failure status, with separate labels for dry-run and break-glass activity. For retention, sinks route logs to log buckets with 1-day to 3650-day retention, to Cloud Storage for low-cost archival, and to BigQuery for SQL analysis. Admin Activity and System Event logs are retained 400 days by default, while Data Access logs are retained 30 days, so critical reads must be exported before expiration and protected with IAM allow policies, uniform bucket-level access, and customer-managed encryption keys when required.

Deployment strategies and success metrics form a closed loop in which limited traffic exposure, real-time health comparison, and automated rollback decide whether a release advances. Scalable and resilient apps (opens in a new tab) depend on progressive delivery to limit blast radius, Cloud Monitoring and Cloud Logging to compare new versus stable versions, and Cloud Deploy to halt or revert promotion. For machine learning, the same loop extends to AI reliability (opens in a new tab) signals such as drift, skew, and output quality.

2.1Success Metrics, SLIs, and Automated Rollback Triggers

Success metrics turn user-perceived reliability into objective promotion gates. A service-level indicator (SLI) is the measured ratio of good events, such as success rate or latency distribution, while a service-level objective (SLO) is the target for that indicator over a time window. Quantified targets make decisions testable, for example 99.9% successful API responses, 95th percentile latency below 300 ms, time to first token below 500 ms for 99% of requests, or harmful-output rate below 0.1%. When the SLI burns its error budget too fast, the deployment is unhealthy even if infrastructure looks normal.

Cloud Monitoring collects deployment health signals by ingesting Google Cloud system metrics, Compute Engine metrics through the Ops Agent, Prometheus metrics through Google Cloud Managed Service for Prometheus, custom metrics through the API or OpenTelemetry, and log-based metrics derived from Cloud Logging. Queries are written with menus or with Monitoring Query Language (MQL) or Prometheus Query Language (PromQL). Cloud Logging adds event context that links an SLI drop to specific errors or deployments. Together they cover the four golden signals: latency including inference latency, traffic such as request rate, error rate including 5xx or failed predictions, and saturation such as CPU, memory, GPU, or TPU utilization.

An automated rollback closes the loop from alert to recovery. An alerting policy defines the metric query, threshold, and notification channels such as email, PagerDuty, Slack, or a Cloud Pub/Sub topic for programmatic response, and each firing creates an incident with status, logs, charts, and duration. Recovery testing (opens in a new tab) follows an ordered guardrail: deploy narrowly to a canary subset, evaluate SLI thresholds, route the alert to halt promotion and restore the stable revision, diagnose with Cloud Trace, Cloud Profiler, and Error Reporting, then restart progressive expansion only after the fix.

2.2Rolling, Blue/Green, Canary, and Feature Flag Traffic Patterns

Progressive deployment releases a new version to a subset of traffic and expands only as telemetry proves success. Google Kubernetes Engine (GKE), Cloud Run, and App Engine each host multiple versions side by side, while Anthos Service Mesh, Traffic Director, and Cloud Load Balancing control how requests split. Feature flags add an application-layer switch so code can deploy without becoming visible.

GKE rolling deployments replace Pods incrementally while the Service sends traffic only to ready Pods, so unready Pods never receive production traffic and failed rollouts can halt early. Blue/green keeps two full environments where blue serves all production traffic while green is validated, then cutover changes the routing rule so traffic flows from the load balancer or Service to green; rollback flows traffic back to blue without redeploying. Canary follows an ordered progression: deploy stable plus canary side by side, split a small share from ingress to canary, evaluate error, latency, and business metrics in Cloud Monitoring, then expand toward canary only on success or return to 100% stable on failure. On Cloud Run and App Engine, splitting is native by revision or version percentage. On GKE, precise splitting depends on Envoy sidecars or proxyless clients receiving policy from the control plane and splitting by weight, header, or cookie. Feature flags isolate exposure differently: new code deploys everywhere, but requests flow from flag evaluation to old or new paths by user, cohort, or percentage, allowing instant disable without rerouting infrastructure.

2.3ML Pipeline Telemetry and Model Serving Signals

ML pipeline telemetry decides when a deployed model is healthy or needs retraining or rollback. Vertex AI endpoints emit per-replica autoscaling signals, Dataflow pipelines emit data-flow and inference signals, and GKE gateways emit serving-distribution signals. Model Monitoring on Vertex AI compares live inputs against a golden dataset, uses TensorFlow Data Validation to establish a baseline schema, and raises alerts that can trigger Vertex AI Pipelines retraining or Model Registry rollback to the last stable version.

Vertex AI online prediction autoscaling adds or removes replicas between a minimum and maximum to keep the selected metric near target. Configuration uses gcloud ai endpoints deploy-model with endpoint ID, model ID, machine type such as n1-standard-4, accelerator type and count, replica bounds, metric name, and threshold. Supported metrics include CPU utilization and GPU duty cycle with 0-100 targets defaulting to 60%, plus DCGM GPU utilization, vLLM KV cache usage, vLLM waiting requests, request count per minute per replica, and Pub/Sub undelivered messages. Because each metric tracks a different bottleneck, a GPU-heavy large language model depends on GPU, DCGM, or vLLM signals, while a request-bound service depends on request count or queue size.

Dataflow ML runs batch and streaming pipelines where data flows from Pub/Sub or Cloud Storage through Apache Beam transforms to inference and downstream processing. The RunInference transform applies local handlers for PyTorch, scikit-learn, TensorFlow, spaCy, Gemma, or Hugging Face models, or remote inference to a Vertex AI endpoint. The MLTransform class keeps training and serving preprocessing consistent, including vocabulary mapping, numeric scaling, text embeddings, and Vertex AI Feature Store enrichment. TensorFlow Model Analysis compares two models in one pipeline, keyed handlers score segments with different versions, and side inputs deliver a new model to a running pipeline without restart.

Choose GKE Inference Gateway when serving depends on LLM-specific state such as KV cache pressure, variable prompt cost, LoRA placement, or gateway filtering. Choose GKE Gateway with Custom Metrics when load is described by one or two application metrics and consistency with Application Load Balancer deployments matters.

Troubleshooting deployments (opens in a new tab) means restoring service to a known-good state by finding which layer failed: configuration, build stage, identity, quota, or network. Infrastructure Manager applies Terraform configurations as versioned deployments, Cloud Build orchestrates pipeline steps, Image Builder customizes images on a temporary worker virtual machine (VM), and Cloud Deploy tracks release progression through rollouts, phases, jobs, and job runs. Systematic diagnosis moves from prerequisites to stage logs to interactive debugging, then to authentication, quota, and observability correlation.

3.1Infrastructure as Code and Pipeline Execution Failures

Infrastructure Manager deployment updates reapply Terraform to correct drift or reach a new desired state. Drift occurs when resources change outside Terraform, such as a console edit, and reapplying the original configuration returns infrastructure to the chosen state. A failed update is often a missing prerequisite rather than syntax. The caller needs roles/config.admin, the service account needs permission for Infrastructure Manager plus resources in both current and new revisions, the Infrastructure Manager API must be enabled, the configuration must not contain sensitive data, quotas must allow the intended resources, and private Git repositories must be connected to Cloud Build. Preview changes before applying to production to expose quota, permission, and dependency problems.

Pipeline stage diagnosis matches the symptom to the logging channel that records that layer. Traffic flows from the customization agent on the worker VM to Cloud Build trace logs and to a Cloud Storage workdir bucket, and the failing step prints a link to its workdir directory.

Interactive debugging changes the failure lifecycle from failed then deleted to failed then paused. By default a failed Image Builder worker VM is deleted immediately, but setting debug: true under infrastructureConfig in imagebuilder.yaml leaves it running for SSH inspection. The ordered workflow is to enable the flag with bounds such as machine type, zone, and instance duration, submit the build and wait for failure, then connect to inspect files and reproduce commands. Resolve Cloud Build timeouts separately by adding time, enlarging the worker, or splitting steps. Authentication and network blockers form the next boundary: enable Service Management and Service Control APIs, check build and node service accounts on 401 or 403 errors, verify VPC firewall rules, network tags, and VPC Service Controls perimeters around Cloud Storage and BigQuery, and narrow scope by asking whether failure is project-wide or isolated and whether it followed a change or traffic spike.

3.2Cloud Deploy Rollout States and Mitigation Operations

Cloud Deploy canary analysis splits traffic between stable and canary versions so metrics can be compared before full rollout. Traffic flows from the load balancer or service mesh to both versions during the canary phase, then only to the new version after advancing to the stable phase. For non-standard strategies, an operator must advance phase to phase, creating an explicit decision point. On first deployment with no running version, Cloud Deploy skips canary phases directly to stable and marks skipped phases and jobs as SKIPPED, which means no comparison occurred rather than a failure.

A rollout tracks overall delivery to a target, a phase tracks one stage, a job tracks one unit of work such as deploy or verify, and a job run tracks one execution attempt. Lower levels propagate upward, so a failed job run fails its job, which fails its phase and rollout.

Rollout states include:

• PENDING: Has not begun; will move to IN_PROGRESS or CANCELED.
• PENDING_APPROVAL: Requires approval before continuing.
• PENDING_RELEASE: Waiting for release render.
• IN_PROGRESS: A job run is processing.
• FAILED: A job failed and was not ignored.
• SUCCEEDED: Finished with no failures.

Phase states include PENDING, IN_PROGRESS, SUCCEEDED, FAILED, ABORTED when a prior phase failed, and SKIPPED for bypassed first-deployment canaries. Job states add IGNORED for an excused failure, DISABLED for an inactive verify job when verification is off, and ABORTED when a prior job blocked execution. Job run states move through IN_PROGRESS, SUCCEEDED, FAILED, TERMINATING, and TERMINATED after user termination. Mitigation acts at different levels: advance a canary after clean metrics, cancel the rollout to stop further phases, terminate a running job run, ignore a known non-blocking failure, or retry a transient failure without creating a new release.

3.3Correlating Metrics, Logs, and Traces to Classify Failures

Diagnosis correlates metrics showing what changed, logs showing what happened, and traces showing where requests slowed across services. Cloud Logging uses aggregated sinks to route entries to a central repository and onward to BigQuery, Cloud Storage, Pub/Sub, or a SIEM, while VPC Flow Logs, firewall rules logging, and load balancer logging make network denials visible alongside API events. Effective log diagnosis follows an ordered lifecycle:

1. Enable logging: Enable audit logging for services in use and define bucket retention.
2. Create log-based counter metrics: Convert matching entries such as Permission Denied errors into time-series data.
3. Configure alerting policies: Open incidents when rates exceed thresholds or high-severity filters match.
4. Create notification channels: Route incidents with Logs Explorer links and remediation steps.

Cloud Monitoring turns those metrics into dashboards and alerts, including VM configuration views, GKE allocation and node-pressure metrics, and firewall insights with allowed and denied ports. Alert quality matters: target Permission Denied errors, unexpected key creations, or firewall changes rather than every informational event, and include query, project, revision, and containment action.

Traffic that fails before reaching the workload points to configuration or infrastructure, while traffic that reaches the workload then throws exceptions points to runtime code. Evidence must be preserved because Admin Activity and System Event logs are retained 400 days while Data Access logs are retained 30 days by default, so critical reads must be exported to BigQuery or Cloud Storage before expiration and protected with Bucket Lock, retention policies, uniform bucket-level access, and restricted IAM access to log configuration.

Key Facts

• Artifact Analysis scan metadata becomes stale after 30 days without an image pull and archived after 90 days stale, with archived metadata available only through the API and refresh after a pull taking up to 24 hours.
• Binary Authorization denies deployment to GKE and Cloud Run when required attestations for builder identity, vulnerability compliance, or promotion criteria are missing.
• Admin Activity audit logs use log name activity and cannot be disabled, while Data Access audit logs use data_access and require Private Logs Viewer for viewing in the _Default bucket.
• Rolling updates replace GKE Pods incrementally, blue/green switches traffic 100% from blue to green for instant cutover, and canary with traffic splitting expands from a small percentage only after telemetry passes.
• Cloud Deploy marks bypassed first-deployment canary phases as SKIPPED, marks phases blocked by prior failures as ABORTED, and allows failed jobs to be marked IGNORED to continue the rollout.