Designing CI/CD Pipelines on Google Cloud

CI/CD of applications and infrastructure is the automated path that moves code and environment definitions from commit to production with testing and security checks at every stage. Cloud Build (opens in a new tab) builds and tests artifacts, Artifact Analysis (opens in a new tab) scans images in Artifact Registry (opens in a new tab), Binary Authorization blocks untrusted images at deploy time, and Cloud Deploy (opens in a new tab) promotes releases through staging to production. Terraform and Cloud Build extend the same model to infrastructure, where Git is the source of truth for desired state.

1.1Rapid and Full Pipelines for Continuous Testing

Continuous testing across rapid and full pipelines means every change is validated automatically before its artifact can be promoted. A Cloud Build trigger starts on commit, then runs linting, static analysis, expert review, and application tests including unit, functional, integration, and load or performance tests, plus Container Structure Tests and Web Security Scanner crawls for cross-site scripting, injection, and outdated libraries.

Rapid pipelines give developers fast feedback, while full pipelines run the complete quality gate that decides whether an image digest is eligible for promotion.

1.2Vulnerability Scanning and Deploy-Time Enforcement

Vulnerability scanning and deploy-time enforcement form a trust boundary between the registry and the runtime that allows only verified images to run. Data flows from Cloud Build to Artifact Registry when the image is pushed, then Artifact Analysis extracts package metadata, compares it to Google Cloud-maintained vulnerability sources, and continuously updates findings by creating new occurrences and deleting invalid ones. An image above the allowed severity fails the gate, although non-applicable findings can be allowlisted.

Binary Authorization enforces the result at deployment to Google Kubernetes Engine (GKE), Cloud Run, and GKE Enterprise by requiring signed attestations for that exact digest, such as built by a specific Cloud Build pipeline, compliant with vulnerability policy, passed quality assurance (QA) testing, or signed off by the product owner. The ordered flow preserves the link from tested digest to deployed digest:

1. Commit and trigger: Developer commits code, which triggers Cloud Build.
2. Build and push: Cloud Build rebuilds the container, pushes it to Artifact Registry, and creates an image digest.
3. Test: Cloud Build runs application and Container Structure Tests.
4. Scan: Cloud Build runs vulnerability scanning.
5. Approve: Cloud Build approves the image only after passing scans and tests.
6. Sign: Binary Authorization signs the image to create the attestation.

Images must be promoted rather than rebuilt between development, QA, and production, with environment-specific configuration read at runtime, so the same immutable digest moves forward.

1.3Progressive Delivery with Immutable Images and Cloud Deploy

Progressive delivery with Cloud Deploy moves the same immutable image through a defined sequence of targets. An immutable image is never changed while running, so any patch requires a new digest and redeployment, and Kubernetes manifests for Deployment, Service, PodMonitoring, VirtualService, DestinationRule, AuthorizationPolicy, Gateway, and Workload Identity Federation mappings stay externalized. Cloud Deploy registers staging and production targets, stores Skaffold rendering sources and rendered manifests in a Cloud Storage bucket, generates a release per source change, deploys to staging first, and promotes the identical digest to production after tests and approvals.

Per-environment logic controls automation, such as auto-deploy to development and manual approval for production, while Skaffold shares rendering logic between local validation with Cloud Code and minikube and production deployment to GKE. Because GKE provides scaling and upgrade logic, teams compose rollout behavior instead of rebuilding rolling or blue-green patterns from scratch.

1.4GitOps Promotion for Terraform Infrastructure

GitOps promotion for Terraform infrastructure makes the Git repository the single source of truth and Cloud Build the engine that reconciles desired state with actual state. Declarative code defines what should exist rather than manual steps, which gives repeatable infrastructure, better rollback, auditing, and policy as code. A push of Terraform code to a tracked branch triggers Cloud Build, which runs Terraform against the matching environment, with traffic flowing from the repository to Cloud Build and then to the target Virtual Private Cloud (VPC) network.

1. Propose in a feature branch: Developer opens a pull request, and the Cloud Build GitHub app links a terraform plan report for review with nothing applied.
2. Promote to development: Merge into the dev branch triggers Cloud Build to run terraform apply to the dev VPC.
3. Promote to production: After validation, merge from dev to prod triggers Cloud Build to apply to the prod VPC.

Shared state in a Cloud Storage bucket gives every execution the same authoritative record, and the Cloud Build service account provides pipeline identity with permissions to read configuration, read and write state, and modify target infrastructure. Preventive guardrails surround the flow with Identity and Access Management (IAM), Organization Policy Service, Tags with IAM Conditions, Policy Controller with Open Policy Agent (OPA), and VPC Service Controls, while the push model centralizes control across many projects and the pull model uses local agents for environment-local reconciliation.

Artifact management with Google Cloud Artifact Registry (opens in a new tab) controls the lifecycle, placement, and promotion of versioned packages and container images. Each format such as Docker, Maven, npm, Python, Apt, and Yum lives in its own repository hierarchy, and Cloud Build, GKE, and Cloud Run all resolve through the same endpoints. Cleanup policies control what stays, while staged repositories and attestations control what moves toward production.

2.1Standard, Remote, and Virtual Repository Structure

Standard, remote, and virtual repositories divide storage, caching, and resolution work for each package format. A standard repository is the system of record for organization-built artifacts, a remote repository is a caching proxy for an external or cross-project upstream such as Maven Central, Docker Hub, or Python Package Index (PyPI), and a virtual repository stores no artifacts but routes requests to its upstreams in priority order.

A virtual repository depends on its upstreams, so upstreams must exist in the same format first. Prioritizing the private standard repository over the remote public cache mitigates dependency confusion, where an attacker publishes a higher-version public package with the same name as a private package.

2.2Cleanup Policies for Lifecycle Management

Automated cleanup policies in Artifact Registry are repository-level rules that delete or preserve versions by age, tag state, and version count. Removing unused, untagged, or stale release-candidate images lowers storage cost and prevents redeployment of old vulnerable software. Policies run in dry-run mode to log candidates or in enforced mode to delete or keep artifacts.

When package_name_prefixes is set on a keep-most-recent-versions policy, only matching packages count toward keep_count. In Terraform, cleanup_policy_dry_run on google_artifact_registry_repository controls enforcement, where true logs without deletion and false enforces deletion.

2.3Secure Promotion and Integrity Enforcement

Secure promotion in Artifact Registry is a staged pipeline that copies or re-tags only validated digests from development to testing to production repositories. Repository-level IAM separates stages so development writers cannot write directly to production, and more attestations are required closer to production. The ordered sequence flows from build to enforcement:

1. Build and pre-push scan: Cloud Build calls On-Demand Scanning on the local image and blocks upload if severity exceeds the threshold.
2. Push to development: The build pushes to development, which triggers automatic scanning by Artifact Analysis (opens in a new tab) with continuous updates.
3. Attach validation metadata: People or automation attach occurrences recording test or scan success.
4. Promote with attestation: The pipeline copies the digest forward only after required attestations are created.
5. Enforce at deploy: Binary Authorization allows only images from trusted registries with required attestations.

Integrity depends on immutable references, attested gates, and supply-chain isolation. Immutable tags or strict SHA-256 digest references such as @sha256:... prevent tag mutation, attestations bind scan and Software Bill of Materials (SBOM) evidence to the digest, and virtual repositories with internal-first priority resist substitution. Continuous Validation watches running Pods in GKE and emits Logging alerts on drift, while breakglass bypasses generate audit logs for security operations.

2.4Scanning Modes, Access Control, and Encryption

Scanning modes, access control, and encryption protect artifacts at rest, in transit, and on the path to runtime. Automatic scanning starts when the Container Scanning API is enabled for the project, after which billing begins and pushes are scanned automatically unless excluded per repository. On-Demand Scanning is a separate manual API for shifting checks earlier into the build.

A shift-left pipeline scans with On-Demand Scanning first, uploads only clean images, and relies on automatic scanning for newly disclosed vulnerabilities. Repository access uses Artifact Registry IAM roles per repository, with allUsers for public and allAuthenticatedUsers for Google-authenticated access, plus IAM Conditions, Cloud Logging audits, and VPC Service Controls perimeters. Encryption defaults to Google-managed keys, while customer-managed encryption keys (CMEK) encrypt a repository with a Cloud Key Management Service key such as projects/KMS-PROJECT/locations/KMS-LOCATION/keyRings/KEY-RING/cryptoKeys/KEY. Repository descriptions and labels are not encrypted, and Security Command Center (opens in a new tab) creates findings only for HIGH or CRITICAL vulnerabilities on images deployed to GKE, Cloud Run, and App Engine.

Deployment to hybrid and multi-cloud environments keeps clusters on-premises, at the edge, and in multiple clouds consistent through centralized desired state and delivery. Google Kubernetes Engine (GKE) (opens in a new tab) clusters are grouped as fleets and synchronized by Anthos Config Management, while Artifact Registry, Cloud Build, and Cloud Deploy provide one build-once, promote-anywhere supply chain. Secrets stay outside artifacts and are injected at runtime.

3.1Declarative GitOps with Config Sync and Fleets

Declarative GitOps with Config Sync and fleets stores desired cluster state as versioned Kubernetes Resource Model (KRM) files and makes every cluster match that state. KRM files define namespaces, role-based access control (RBAC), and workloads through the Kubernetes API, so the same manifest produces the same result in cloud and private environments. Config Sync (opens in a new tab) pulls declared manifests from the source of truth to each fleet member and continuously reapplies them to correct drift such as manually edited Deployments or changed bindings.

1. Declare: Authors commit KRM manifests and Kustomize customizations to versioned source.
2. Distribute: Config Sync pulls desired state from the central management plane to each cluster.
3. Apply: Each cluster applies manifests through its Kubernetes API server.
4. Reconcile: Agents compare live state to desired state and remediate drift.

A GKE fleet is a logical grouping that enables sameness, where objects with the same name across clusters are treated as the same thing, plus shared bases with small overlays for image tags, replicas, or load balancer behavior. Administrative access flows through the Connect gateway, which fetches credentials for multiple clusters with Google Cloud identity.

3.2Secure Build and Artifact Supply Chain for Hybrid Delivery

The secure build and artifact supply chain for hybrid delivery centralizes dependencies, proves build integrity, and enforces policy before any cluster admits an image. Artifact Registry exposes direct and transitive dependencies through remote repositories that cache external sources and virtual repositories that consolidate same-format repositories behind one endpoint with internal-first search order. Artifact Analysis scans base images and language packages, generates SBOMs, supports Vulnerability Exploitability eXchange (VEX) statements, and continuously updates occurrences as new vulnerabilities appear.

Cloud Build runs on Google Cloud infrastructure with granular IAM, VPC Service Controls, and isolated ephemeral environments, and supports Supply-chain Levels for Software Artifacts (SLSA) Level 3 builds with authenticated provenance including digests, source locations, toolchain, steps, and duration. Binary Authorization then checks attestations at deploy time to verify build origin, vulnerability policy compliance, and promotion criteria from development to QA. Cloud Deploy promotes the same signed digest to GKE, GKE Enterprise, attached clusters, and Cloud Run with approvals, audit, metrics, and security insights, using trusted tools and private ingress or egress to limit exfiltration.

3.3Fleet Registration and Sequenced Rollouts

Fleet registration and sequenced rollouts control which clusters belong together and in what order changes reach them. The fleet is managed by the Fleet service, also called Hub, through a single fleet host project that uplevels observability from individual clusters to groups. Organizations use multiple clusters to separate production from non-production, isolate failure domains across zones and regions, and enable team ownership with central oversight.

The Connect Agent traverses NATs, proxies, and VPNs so API servers need no public IPs, and the Connect gateway provides consistent command-line access without direct IP connectivity. Team scopes bind subsets of clusters to application teams for access control, observability, and ordered upgrades from non-production to broad production, which limits blast radius. Lightweight memberships still permit sequenced upgrade rollouts but do not support fleet-level features, default logs, default metrics, or Fleet Workload Identity Federation.

3.4Secrets and Traffic Control Across Environments

Secrets and traffic control across environments keep credentials out of delivery artifacts and expose only chosen APIs between networks. Kubernetes Secrets resemble ConfigMaps but are not secure by default without encryption, so secrets for Cloud Deploy applications must be generated, managed, and rotated outside Cloud Deploy and referenced at runtime.

Global routing follows a gated egress pattern where traffic flows from the Google Cloud VPC to a remote gateway through workload placement, hybrid extension over overlap-free RFC 1918 space, IP allowlist filtering with stateful return rules, and inspection by firewall rules, network virtual appliance (NVA), or Cloud Next Generation Firewall Enterprise. Traffic to private APIs initiates only from Google Cloud, private systems are not directly reachable, and pod-to-pod traffic within clusters is open by default and must be restricted with Cloud Service Mesh or Kubernetes network policies.

CI/CD pipeline triggers in Cloud Build (opens in a new tab) are rules that start builds automatically when repository or external events occur. Each trigger connects to a source, filters events by branch, tag, or pull request, and runs a build configuration under a designated service account. This design lets one repository fan out to different pipelines without manual invocation.

4.1Version Control Triggers and Branch Filtering

Version Control System (VCS) triggers start builds when repository activity matches the trigger event and filter. The trigger connects to GitHub, GitHub Enterprise, Bitbucket Cloud, or Cloud Source Repositories, and only matching events cross the control boundary into build execution.

Automated triggers match branches and tags with regular expressions in BRANCH_PATTERN and TAG_PATTERN, such as main or release-.*, while manual triggers require a literal branch, tag, or commit SHA. Second-generation triggers use Developer Connect host connections, and the repository region must match the trigger region. If the build uses a private pool, the trigger region must also match the private pool region; otherwise the build runs in the default pool in the trigger region.

4.2Build Execution, Substitution, and Event-Driven Workflows

Build execution, substitution, and event-driven workflows define what runs after a trigger fires and how one definition adapts to different events. The configuration can be a cloudbuild.yaml or cloudbuild.json file in the repository or inline content, a Dockerfile with directory and image name, or a buildpacks directory, and the source checkout is a shallow clone of only the triggering commit. Each build step runs in a Docker container, and different triggers can invoke different cloudbuild.yaml routines from the same repository.

Substitution variables map event metadata into the build so the same definition tags and traces artifacts differently per invocation, including $COMMIT_SHA, $REPO_NAME, $BRANCH_NAME, $TAG_NAME, and $REVISION_ID, while $PROJECT_ID and $BUILD_ID are available outside triggers. A common pattern tags images with $COMMIT_SHA to link each Artifact Registry image to its source event. Event sources extend beyond pushes to data-arrival signals, such as BigQuery ingestion triggering retraining or Cloud Logging filters detecting new data or skew, with the ordered flow from source copy under /workspace through test, analysis, image build and publish, reference update, workflow compile to pipeline.json, publish, and optional execution.

4.3Secure Trigger Identity with Least Privilege

Secure trigger identity runs each automated build under a dedicated user-specified service account with only required permissions. The flow moves from source event to trigger to build runtime to Google Cloud resources, where the assigned account reads source, writes artifacts, and calls deployment services. Default identities such as PROJECT_NUMBER@cloudbuild.gserviceaccount.com and PROJECT_NUMBER-compute@developer.gserviceaccount.com are shared and often broader than production needs, especially the Compute Engine default account.

Least privilege grants resource-level roles instead of project-level access, with IAM Conditions for attribute-based invocation, principal access boundary policies to cap effective access, Privileged Access Manager for just-in-time elevation, and bucket, managed-folder, and BigQuery row or column controls. Creating and editing triggers requires roles/cloudbuild.builds.editor, acting as another runtime identity requires roles/iam.serviceAccountUser on that account, and production promotion adds approvals, Artifact Analysis scanning, SLSA provenance, and Binary Authorization attestations before GKE or Cloud Run admits the image.

Configuring deployment processes means building approval gates that pause promotion until required checks and authorizations succeed. On Google Cloud, gates combine Cloud Deploy (opens in a new tab) targets and approvals, Binary Authorization attestations, Artifact Analysis scans, and IAM separation of duties. Development stays fast while staging and production require verified artifacts and independent approvers.

5.1Risk-Based Approval Gates and Environment Promotion

Risk-based approval gates are integrity controls that prevent unauthorized, untested, or untrusted changes from reaching environments. The pipeline enforces the same verified process for every change, including code review, automated tests, static analysis, policy as code, and limits on simultaneous changes to narrow review scope. Environment-tiered policies make approval stricter as criticality rises, typically promoting passing builds to preproduction for end-to-end tests and updating production only after those tests and approvals succeed.

Cloud Deploy automates delivery to GKE, GKE Enterprise, and Cloud Run in a defined sequence with one-click approvals and rollbacks, audit, metrics, and security insights. Cloud Build provides isolated ephemeral builds with SLSA Level 3 provenance recording digests, source locations, toolchain, steps, and duration. The SLSA framework organizes maturity across source, build, and provenance, so only artifacts that pass the checks for that environment can deploy.

5.2Cloud Deploy Approvals with Pub/Sub and IAM Separation

Cloud Deploy approvals with Pub/Sub and IAM separation determine who can request promotion and who can authorize it. A target representing a high-risk environment sets requireApproval: true, which places the rollout in a pending approval state until an authorized identity approves or rejects it. Only an identity with roles/clouddeploy.approver can clear the gate with gcloud deploy rollouts approve, so deployment and approval permissions can belong to different people or systems.

External approval notifications use the clouddeploy-approvals Pub/Sub topic to connect delivery to ticketing or review boards. Data flows from Cloud Deploy to Pub/Sub to the workflow system, then back to Cloud Deploy as an approve or reject command:

1. Require approval on the target: Add requireApproval: true to the target definition.
2. Subscribe to approval events: Subscribe the workflow system to clouddeploy-approvals.
3. Start organizational review: On a message with "Action": "Required" and rollout reference Resource: projects/.../locations/.../deliveryPipelines/.../releases/.../rollouts/..., start routing, escalation, and notifications.
4. Return the decision: Issue gcloud deploy rollouts approve ROLLOUT --delivery-pipeline=PIPELINE_NAME --region=REGION --release=RELEASE_NAME or the matching reject command.

5.3Automated Quality Checks and Pipeline Integrity Protection

Automated quality checks and pipeline integrity protection block promotion until scans, tests, and policy evaluations succeed and record verifiable evidence. Binary Authorization (opens in a new tab) attaches attestations proving a required process completed and checks them at deployment to GKE and Cloud Run, including build origin from a trusted pipeline, vulnerability policy compliance, and promotion criteria from development to QA. Because enforcement happens at the target platform, compromised pipeline configuration alone cannot run untrusted code when policy requires valid attestations.

Artifact Analysis (opens in a new tab) supplies continuous evidence by scanning on upload to Artifact Registry, extracting system package data, monitoring stored metadata against new vulnerability sources, updating metadata, creating new occurrences, and deleting invalid ones. A previously clean image can therefore fail approval later when new data arrives. In-pipeline controls before manual approval add static analysis, policy as code, and change limits, while pipeline integrity follows the Biba model where the pipeline must not consume lower-integrity inputs or write to higher-integrity resources. Teams verify signed artifacts with Pretty Good Privacy (PGP) or Authenticode, verify hashes for unsigned artifacts, obtain Terraform, kubectl, Docker, and operating systems from trusted sources, rebuild regularly, and run on shielded virtual machines (VMs).

Key Facts

• Cloud Build creates and validates the immutable container image and calls Cloud Deploy, while Cloud Deploy creates the release and promotes that same image digest from staging to production.
• Binary Authorization allows deployment only for images with required attestations, and continuous validation only logs policy violations for already-running Pods to Cloud Logging.
• Automatic vulnerability scanning continuously updates findings for up to 30 days after push, while On-Demand Scanning results expire after 48 hours and are never updated.
• A push to a Terraform feature branch runs terraform plan for review only, while a merge into the dev or prod branch runs terraform apply to that environment using shared state in a Cloud Storage bucket.
• Cloud Deploy pauses promotion on any target configured with requireApproval: true until an identity with roles/clouddeploy.approver runs gcloud deploy rollouts approve.