Building CI/CD Pipelines with Cloud Build and Deploy

1.1CI/CD Path From Commit to Production

The CI/CD path from commit to production is the automated sequence that moves application code and infrastructure definitions through build, test, scan, and promotion. Cloud Build builds and tests artifacts, Artifact Registry stores versioned images, Artifact Analysis scans images for vulnerabilities, Binary Authorization blocks untrusted images at deploy time, and Cloud Deploy promotes the same immutable digest from staging to production. Terraform extends the same model to infrastructure, where Git is the source of truth for desired state and Cloud Build reconciles that state with the target Virtual Private Cloud (VPC) network.

Continuous testing uses two pipeline types that trade speed for completeness. A Cloud Build trigger starts on commit and runs linting, static analysis, unit, functional, integration, and load tests, plus Container Structure Tests and Web Security Scanner checks for cross-site scripting, injection, and outdated libraries. Only images that pass the full gate are eligible for promotion, and images must be promoted rather than rebuilt so the same digest moves from development to quality assurance (QA) to production with environment configuration read at runtime.

Vulnerability scanning and deploy-time enforcement form a trust boundary between the registry and the runtime. Data flows from Cloud Build to Artifact Registry on push, then Artifact Analysis extracts package metadata, compares it to Google Cloud-maintained sources, and continuously creates new occurrences and deletes invalid ones. The ordered flow preserves the link from tested digest to deployed digest:

1. Commit and trigger: Developer commits code, which triggers Cloud Build.
2. Build and push: Cloud Build rebuilds the container, pushes it to Artifact Registry, and creates an image digest.
3. Test: Cloud Build runs application and Container Structure Tests.
4. Scan: Cloud Build runs vulnerability scanning.
5. Approve: Cloud Build approves the image only after passing scans and tests.
6. Sign: Binary Authorization signs the image to create the attestation.

1.2Artifact Lifecycle and Promotion in Artifact Registry

Artifact lifecycle and promotion in Google Cloud Artifact Registry controls where versioned packages live, how long they are kept, and what moves toward production. Each format such as Docker, Maven, npm, Python, Apt, and Yum lives in its own repository hierarchy, and Cloud Build, Google Kubernetes Engine (GKE), and Cloud Run all resolve through the same endpoints. Cleanup policies control what stays, while staged repositories and attestations control what is promoted.

Standard, remote, and virtual repositories divide storage, caching, and resolution work. A standard repository is the system of record for organization-built artifacts, a remote repository is a caching proxy for an external upstream such as Maven Central, Docker Hub, or Python Package Index (PyPI), and a virtual repository stores no artifacts but routes requests to its upstreams in priority order. Traffic flows from the client to the virtual repository to the first upstream that satisfies the request. Prioritizing the private standard repository over the public cache mitigates dependency confusion, where an attacker publishes a higher-version public package with the same name as a private package.

Automated cleanup policies are repository-level rules that delete or preserve versions by age, tag state, and version count. Conditional delete removes matching artifacts by older_than duration, tag state, and package_name_prefixes, conditional keep preserves protected packages, and keep-most-recent-versions keeps only N versions per package defined by keep_count. Policies run in dry-run mode to log candidates or in enforced mode to delete, and in Terraform cleanup_policy_dry_run on google_artifact_registry_repository selects the mode where true logs without deletion and false enforces deletion.

1.3Hybrid and Multi-Cloud Delivery With Fleets and Config Sync

Hybrid and multi-cloud delivery keeps clusters on-premises, at the edge, and in multiple clouds consistent through centralized desired state. Google Kubernetes Engine (GKE) clusters are grouped as fleets and synchronized by Anthos Config Sync, while Artifact Registry, Cloud Build, and Cloud Deploy provide one build-once, promote-anywhere supply chain. Secrets stay outside artifacts and are injected at runtime.

Declarative GitOps with Config Sync stores desired cluster state as versioned Kubernetes Resource Model (KRM) files that define namespaces, role-based access control (RBAC), and workloads. Config Sync pulls declared manifests from the source of truth to each fleet member and continuously reapplies them to correct drift such as manually edited Deployments. The ordered reconciliation is declare in versioned source, distribute from the central management plane to each cluster, apply through each Kubernetes API server, then reconcile live state to desired state. A GKE fleet enables sameness, where objects with the same name across clusters are treated as the same thing, with shared bases and small overlays for image tags or replicas.

Fleet registration and sequenced rollouts control membership and update order. The Fleet service, also called Hub, manages membership through a single fleet host project. GKE on Google Cloud requires explicit registration, Google Distributed Cloud on-premises registers automatically at creation, and GKE on AWS and Azure registers automatically with a host project. The Connect Agent traverses NATs, proxies, and VPNs so API servers need no public IPs, and the Connect gateway provides consistent access without direct IP connectivity. Team scopes bind subsets of clusters to application teams for ordered upgrades from non-production to broad production, which limits blast radius.

1.4Pipeline Triggers, Substitutions, and Secure Identity

Pipeline triggers in Cloud Build are rules that start builds automatically when repository or external events occur. Each trigger connects to a source, filters events by branch, tag, or pull request, and runs a build configuration under a designated service account. This design lets one repository fan out to different pipelines without manual invocation.

Version Control System (VCS) triggers start builds only when the event and filter match. Push to a branch starts continuous builds for each change, push of a new tag starts release builds for tagged versions, and pull request activity starts validation builds before merge. Automated triggers match branches and tags with regular expressions in BRANCH_PATTERN and TAG_PATTERN, while manual triggers require a literal branch, tag, or commit SHA. Second-generation triggers use Developer Connect host connections, and the repository region must match the trigger region. If the build uses a private pool, the trigger region must also match the private pool region.

Build execution defines what runs after a trigger fires. The configuration can be a cloudbuild.yaml or cloudbuild.json file, a Dockerfile with directory and image name, or a buildpacks directory, and the source checkout is a shallow clone of only the triggering commit. Each build step runs in a Docker container. Substitution variables map event metadata into the build, including $COMMIT_SHA, $REPO_NAME, $BRANCH_NAME, $TAG_NAME, and $REVISION_ID, while $PROJECT_ID and $BUILD_ID are available outside triggers. A common pattern tags images with $COMMIT_SHA to link each Artifact Registry image to its source event. Secure identity runs each build under a dedicated user-specified service account with only required permissions, avoiding the shared Cloud Build service account and the broader Compute Engine default service account.

1.5Approval Gates and Deployment Configuration

Approval gates and deployment configuration pause promotion until required checks and authorizations succeed. Gates combine Cloud Deploy targets and approvals, Binary Authorization attestations, Artifact Analysis scans, and Identity and Access Management (IAM) separation of duties. Development stays fast while staging and production require verified artifacts and independent approvers.

Risk-based gates make approval stricter as environment criticality rises, typically promoting passing builds to preproduction for end-to-end tests and updating production only after those tests and approvals succeed. Cloud Deploy automates delivery to GKE, GKE Enterprise, and Cloud Run in a defined sequence with one-click approvals and rollbacks, audit, metrics, and security insights. Cloud Build provides isolated ephemeral builds with Supply-chain Levels for Software Artifacts (SLSA) Level 3 provenance recording digests, source locations, toolchain, steps, and duration.

Cloud Deploy approvals with Pub/Sub and IAM separation determine who can request promotion and who can authorize it. A target with requireApproval: true places the rollout in a pending approval state until an identity with roles/clouddeploy.approver approves or rejects it with gcloud deploy rollouts approve. External review uses the clouddeploy-approvals Pub/Sub topic. Data flows from Cloud Deploy to Pub/Sub to the workflow system, then back to Cloud Deploy as an approve or reject command for that specific rollout, release, pipeline, and region.

2.1Deployment Auditing From Build to Approval

Deployment auditing from build to approval proves where an artifact came from, how it was built, where it was released, and who approved the change. Artifact Registry stores images and records lineage, Cloud Build creates verifiable build history with step logs and SLSA Level 3 provenance, Cloud Deploy creates ordered release history across targets, and Cloud Audit Logs records who invoked each action. Together they provide operational visibility and compliance evidence for supply chain security.

Artifact provenance and vulnerability scanning proves origin and safety before promotion. Artifact Analysis extracts package inventory on push and records vulnerabilities as metadata tied to the digest, while Artifact guard evaluates images against centralized policies during build, storage, and runtime. Enabling scanning requires project-level enablement of the Artifact Registry and Container Scanning APIs plus per-repository enablement. Viewing requires Artifact Registry Reader, Container Analysis Occurrences Viewer, and Container Analysis Notes Viewer.

Scan metadata follows an active-to-stale-to-archived lifecycle driven by pulls. The service refreshes scans while the image has been pulled within the last 30 days. After 30 days without a pull, results become stale and stop updating. After 90 more days stale, metadata becomes archived and can be evaluated only through the API. Pulling a stale or archived image triggers a re-scan that refreshes within up to 24 hours.

2.2Progressive Strategies and Success Metrics

Progressive strategies and success metrics form a closed loop in which limited traffic exposure, health comparison, and automated rollback decide whether a release advances. GKE, Cloud Run, and App Engine each host multiple versions side by side, while Anthos Service Mesh, Traffic Director, and Cloud Load Balancing control how requests split. Feature flags add an application-layer switch so code can deploy without becoming visible.

Success metrics turn reliability into promotion gates. A service-level indicator (SLI) is the measured ratio of good events such as success rate or latency distribution, while a service-level objective (SLO) is the target over a time window. Cloud Monitoring ingests system metrics, Compute Engine metrics through the Ops Agent, Prometheus metrics through Google Cloud Managed Service for Prometheus, custom metrics, and log-based metrics, queried with menus or with Monitoring Query Language (MQL) or Prometheus Query Language (PromQL). Cloud Logging links an SLI drop to specific errors. An alerting policy defines the query, threshold, and notification channels such as email, PagerDuty, Slack, or Pub/Sub, and each firing creates an incident.

GKE rolling deployments send traffic only to ready Pods, so unready Pods never receive production traffic. Blue/green keeps two full environments where blue serves all traffic while green is validated, then cutover moves traffic to green and rollback moves it back to blue without redeploying. Canary deploys stable plus canary side by side, splits a small share to canary, evaluates error, latency, and business metrics, then expands on success or returns to 100% stable on failure. On GKE, precise splitting depends on Envoy sidecars or proxyless clients receiving policy from the control plane.

2.3Troubleshooting Rollouts and Correlating Signals

Troubleshooting rollouts restores service to a known-good state by finding which layer failed: configuration, build stage, identity, quota, or network. Infrastructure Manager applies Terraform configurations as versioned deployments, Cloud Build orchestrates pipeline steps, and Cloud Deploy tracks release progression through rollouts, phases, jobs, and job runs. Systematic diagnosis moves from prerequisites to stage logs to interactive debugging, then to authentication, quota, and observability correlation.

A rollout tracks overall delivery to a target, a phase tracks one stage, a job tracks one unit of work such as deploy or verify, and a job run tracks one execution attempt. Lower levels propagate upward, so a failed job run fails its job, which fails its phase and rollout. Rollout states include PENDING, PENDING_APPROVAL, PENDING_RELEASE, IN_PROGRESS, FAILED, and SUCCEEDED. Phase states add ABORTED when a prior phase failed and SKIPPED for bypassed first-deployment canaries. Job states add IGNORED for an excused failure and DISABLED when verification is off. Mitigation acts at different levels: advance a canary after clean metrics, cancel the rollout, terminate a running job run, ignore a known non-blocking failure, or retry a transient failure without a new release.

Diagnosis correlates metrics showing what changed, logs showing what happened, and traces showing where requests slowed. Traffic that fails before reaching the workload points to configuration or infrastructure, while traffic that reaches the workload then throws exceptions points to runtime code. Effective log diagnosis follows enable logging with retention, create log-based counter metrics, configure alerting policies, then create notification channels. Evidence must be preserved because Admin Activity and System Event logs are retained 400 days by default while Data Access logs are retained 30 days, so critical reads must be exported to BigQuery or Cloud Storage before expiration.

3.1Envelope Encryption and Customer-Managed Keys

Envelope encryption and customer-managed keys separate the key that encrypts data from the key that encrypts other keys. Cloud Key Management Service (Cloud KMS) holds the key encryption key (KEK) and never exposes raw KEK material, while a per-object data encryption key (DEK) encrypts the payload and travels wrapped alongside it in Secret Manager, Cloud Storage, or another CMEK-integrated service. Reading the payload therefore requires both permission on the resource and permission to unwrap the DEK with the KEK.

The flow starts when a CMEK-integrated service generates a DEK, uses the DEK to encrypt the value with AES-256, then sends the DEK to Cloud KMS where the customer-managed encryption key (CMEK) encrypts it. For decryption, the service sends the wrapped DEK back to Cloud KMS, Cloud KMS decrypts the DEK with the CMEK, and the service uses the unwrapped DEK in memory. All CMEK keys use symmetric ENCRYPT_DECRYPT purpose with GOOGLE_SYMMETRIC_ENCRYPTION, and each key supports unlimited versions where the primary version encrypts new DEKs and older enabled versions decrypt older data. If the CMEK is disabled, destroyed, or its roles/cloudkms.cryptoKeyEncrypterDecrypter grant to the service agent is revoked, protected data becomes unreadable.

3.2Secret Storage, Replication, and Keyless Access

Secret storage, replication, and keyless access controls where sensitive configuration lives and how workloads reach it without long-lived keys. Secret Manager is the IAM-controlled store for secrets, while Cloud KMS provides customer-controlled keys that protect data at rest. Centralized projects for each service let a dedicated team own creation, rotation, and auditing.

Replication policy controls where payloads are stored and served from. Automatic replication lets Google Cloud choose serving locations and is the default for availability and performance, while user-managed replication selects locations via API for strict residency requirements. The Cloud KMS key ring must reside in a location compatible with those replicas. Least-privilege access grants CI/CD service accounts only required secret accessor roles, and VPC Service Controls creates perimeters around Secret Manager and Cloud KMS to help prevent exfiltration. Organization Policy constraint constraints/gcp.restrictNonCmekServices requires CMEK protection for listed services, so resource creation without a customer-managed key is denied.

Workload Identity Federation is a keyless method that lets workloads outside Google Cloud access Google Cloud APIs without long-lived service account keys. The external workload presents a credential from its own external identity provider (IdP), and Google Cloud exchanges it via Security Token Service (STS) for short-lived credentials. The ordered exchange is obtain the native credential such as GitHub OpenID Connect (OIDC) token or AWS temporary credential, exchange it with STS referencing the pool and provider, then call the target API with OAuth 2.0 tokens that expire by default after 1 hour. A workload identity pool is the IAM trust boundary, the provider describes a single IdP with issuer and verification method, attribute mapping translates external claims into federated attributes, and an attribute condition restricts which identities can federate.

3.3Build-Time Provisioning Versus Runtime Injection

Build-time provisioning versus runtime injection differs by where the secret resolves and how it reaches the workload. Cloud Build consumes secrets during the build, while services on GKE, Cloud Run, or Vertex AI Pipelines fetch secrets after deployment. Build-time provisioning makes a secret available to an ephemeral build step, while runtime injection makes the live value available to a running workload without baking it into artifacts.

Build-time provisioning delivers a secret version to specific build steps through the availableSecrets block after all build steps in cloudbuild.yaml. The configuration declares a mapping of versionName such as projects/PROJECT/secrets/SECRET/versions/VERSION to env variable name, scopes access per step with secretEnv and entrypoint: bash with args: -c, then consumes the variable without persisting it. The build service account must hold Secret Manager Secret Accessor on the specific secret, and both the Cloud Build and Secret Manager APIs must be enabled.

Runtime injection keeps secrets out of deployment artifacts. The running Pod or service authenticates with Application Default Credentials, Workload Identity on GKE, or workload identity federation on AWS or Azure, then reads a specific secret version via direct API access, native Cloud Run bindings, or Secrets Store CSI driver mounts for GKE. Direct API access avoids expanding the trust boundary to files or environment variables.

4.1Automatic and On-Demand Vulnerability Scanning

Automatic and on-demand vulnerability scanning in Artifact Analysis detects vulnerabilities, stores security metadata, and feeds deployment decisions. Artifact Registry provides built-in detection points where images are scanned on push and continuously monitored afterward. Together they connect detection in the registry to enforcement in the pipeline and visibility in operations.

Automatic scanning scans a new image when the image is pushed to a Docker repository and then keeps findings current through continuous analysis. The on-push scan extracts operating system and language packages, identifies dependencies and licenses, checks for malicious packages from the OpenSSF Malicious Packages Repository, and produces a vulnerability result that collects all occurrences for that image. Scanning is keyed to image digest, so an image is scanned only once and tag changes never trigger a new scan. Continuous analysis ingests new vulnerability data multiple times each day and updates, creates, or deletes occurrences. Findings stay updated only while the image has been pulled within 30 days; after 30 days without a pull metadata becomes stale, and after more than 90 days stale it becomes archived and API-only. Scanning applies only to Linux-based images and prefers Linux amd64 in a manifest list.

On-demand scanning is a manual scan started with the gcloud command-line interface (CLI) against a local or registry image without first pushing. Results are available for up to 48 hours and are never updated. Because on-demand scanning is point-in-time and automatic scanning is continuous, on-demand scanning blocks risky images early in Cloud Build while automatic scanning catches newly disclosed vulnerabilities in stored images.

4.2Supply Chain Security With SLSA and Binary Authorization

Supply chain security with SLSA and Binary Authorization proves where artifacts came from, how they were built, and whether they are safe to deploy. Supply-chain Levels for Software Artifacts (SLSA) measures build-process maturity, Artifact Analysis provides vulnerability truth, and Binary Authorization enforces admission based on signed evidence. Together they move trust from mutable tags to immutable digests with verifiable provenance.

A secure build platform is the foundation for a higher SLSA level. Cloud Build runs builds on Google Cloud infrastructure with granular IAM, VPC Service Controls, and isolated ephemeral environments provisioned for one build then discarded. Cloud Build supports SLSA Level 3 builds for container images and generates authenticated, non-falsifiable provenance including digests, source locations, toolchain, steps, and duration. Vulnerability scanning and software composition analysis (SCA) inventories direct and transitive dependencies, tracks Software Bill of Materials (SBOM) and Vulnerability Exploitability eXchange (VEX) statements, and uses lock files to determine exact dependencies.

Binary Authorization is the deploy-time control that admits only images whose attestations satisfy policy before they become running resources on GKE, Cloud Run, or App Engine. The ordered gate preserves the chain from evidence to admission:

1. Scan: Artifact Analysis scans the image and stores occurrences for that digest.
2. Verify and sign: An authorized attester checks results and creates a signed attestation linked to the immutable SHA-256 digest.
3. Enforce: The admission controller sends the deploy request to Binary Authorization, which verifies signatures before Pod creation.
4. Monitor: Continuous Validation watches active Pods and generates Logging alerts if running containers become non-compliant.

4.3Environment IAM, Network Context, and Audit

Environment IAM, network context, and audit controls who can act on development, testing, and production. IAM allow policies set high in the hierarchy flow downward to every child, while environment folders add only the narrow bindings each tier needs. Production stays strict with approved identities and secure contexts, while development stays flexible without risking production.

Environment isolation uses folders inside one organization to host development, testing, and production for the same application. Policies set at the organization or parent folder inherit to every child, while policies on an environment folder apply only to projects under that folder. A separate staging organization tests organization-wide change with mirrored structure and policies, not daily development work. Least privilege scales with risk: basic roles such as Owner are avoided in production because they include thousands of permissions, while test projects may accept broader roles for velocity. Each component uses its own service account granted only needed permissions at the smallest scope, such as Publisher on a single Pub/Sub topic.

Delivery controls restrict pipeline actions by resource and context. An IAM policy binds roles such as roles/clouddeploy.developer or roles/clouddeploy.operator to a single delivery pipeline or target with setIamPolicy, and IAM Conditions evaluate resource name or time before a binding takes effect. Separation of duties grants promote and approve rights to different principals on the production target. Default IAM ignores geolocation and source IP, so VPC Service Controls adds the network boundary by placing projects inside a perimeter and allowing only expected pipeline paths via ingress rules. Auditing records every setIamPolicy() call in Cloud Audit Logs, and Policy Intelligence tools predict impact: IAM Recommender suggests narrower roles, Policy Simulator tests whether a change breaks access, and Policy Analyzer shows who can do what today.

Key Facts

• Cloud Build builds and tests artifacts and Cloud Deploy promotes the same immutable digest from staging to production, while Binary Authorization admits only digests with required attestations and continuous validation only logs violations for already-running Pods.
• Automatic scanning triggers once per image digest on push and updates continuously only while the image was pulled within 30 days, while on-demand scanning gives a point-in-time result for 48 hours with no updates.
• A Cloud Deploy target with requireApproval: true stays in pending approval until an identity with roles/clouddeploy.approver runs rollouts approve for that rollout, release, pipeline, and region.
• Envelope encryption uses a per-object DEK to encrypt data and a customer-owned KEK in Cloud KMS to wrap the DEK, with a version lifecycle from enabled to disabled to scheduled for destruction to destroyed.
• Environment isolation uses folders for development, testing, and production inside one organization with inherited IAM policies, while VPC Service Controls enforces network context that IAM ignores by default.