Designing CI/CD Architecture for Hybrid Clouds

Continuous integration (CI) with Cloud Build (opens in a new tab) is the automated practice of testing every commit, building a container image, and pushing a versioned artifact to Artifact Registry. A cloudbuild.yaml file declares an ordered list of steps that share source code and a workspace, and a build trigger starts that file when a repository event occurs. Data flows from the source repository to the trigger, from the trigger through the build steps in order, and from the final steps to deployment targets.

1.1Efficient cloudbuild.yaml Steps and Builders

A cloudbuild.yaml build configuration in Cloud Build defines work as an ordered sequence of steps that share the build workspace. Each step names a builder container image, a unique id, an optional entrypoint, args passed to the container, and optional env values. Data flows from the fetched source into the first step and then from step to step in declared order, so a later step can use files created by an earlier step.

The following builders divide work by tool:

A minimal CI pipeline runs three steps in strict order:

1. Test: Run tests in python:3.13-slim, for example with entrypoint: /bin/sh.
2. Build: Run docker build in the Docker builder and tag the image with the commit SHA.
3. Push: Run docker push in the Docker builder to store the tagged image in Artifact Registry.

An extended pipeline that connects integration to delivery adds manifest steps after the push. First a clone step checks out the candidate branch of an environment repository, then a generate step renders kubernetes.yaml from a template with the new project ID and commit SHA, and only after that a push-manifest step commits and pushes to origin candidate.

1.2Build Triggers, Sources, and Testing

A Cloud Build trigger binds a connected source repository, an event filter, a configuration file path, substitution values, and a service account into one repeatable pipeline. Sources include Cloud Source Repositories, GitHub, GitLab, and Bitbucket, linked through Cloud Build repositories or Developer Connect connections. Only pushes that match the branch filter, such as the main branch, start a build, and the build can read the matched branch through $BRANCH_NAME.

Creating a production trigger follows an ordered workflow:

1. Open Triggers: Open the Cloud Build Triggers page and select a region.
2. Connect source: Connect the Git repository if it is not already linked.
3. Name and filter: Enter a trigger name, select an event, repository, and branch.
4. Point to config: Select the cloudbuild.yaml path, such as /cloudbuild.yaml.
5. Set substitutions: Add required user substitution values for the environment.
6. Assign identity: Select a custom least-privilege service account and create the trigger.

Testing validates the full path from source event to downstream effect. Committing and pushing cloudbuild.yaml to the monitored branch starts a build that appears in Build History, and success is confirmed when downstream systems show the expected result, such as a new workflow compilation (opens in a new tab) marked live and a new invocation in execution logs.

1.3Substitutions for Reusable Configurations

Substitutions in Cloud Build are variables replaced at build time that keep one cloudbuild.yaml reusable across projects and environments. Default substitutions such as $PROJECT_ID, $SHORT_SHA, and $COMMIT_SHA are replaced automatically, so the same Build and Push steps produce a correctly namespaced and uniquely tagged image on every commit. Custom substitutions start with an underscore, are declared in a substitutions: block, and are supplied by the trigger, for example $_REGION, $_AR_REPO_NAME, $_AR_IMAGE_NAME, and $_IMAGE_DIR.

Inside bash steps, references use $$ escaping such as $${_PROJECT_ID} so Cloud Build substitutes first and bash expands second. Tagging with the short commit SHA ties each stored image to the exact commit that produced it, such as us-central1-docker.pkg.dev/$PROJECT_ID/my-repository/hello-cloudbuild:$SHORT_SHA. That tag becomes the traceable link written into Kubernetes manifests for deployment.

1.4Build Artifacts, Scanning, and Least-Privilege Identity

Build artifacts in Cloud Build are the versioned outputs kept for deployment, and container images are stored in Artifact Registry repositories. The image path encodes region, project, repository, image, and tag, and distribution depends on identity: the Cloud Build service account needs roles/artifactregistry.writer to push. User-specified service accounts limit each build to only needed roles but must store build logs (opens in a new tab) in Cloud Logging or a user-created bucket, not the default logs bucket.

Security is built into the CI path from artifact to deployment. Artifact Analysis scans images stored in Artifact Registry and shows Supply-chain Levels for Software Artifacts (SLSA) level, vulnerabilities, Vulnerability Exploitability eXchange (VEX) status, software bill of materials (SBOM), and build details in the Security insights panel. Binary Authorization then blocks deployment unless a Cloud Key Management Service (Cloud KMS)-signed attestation proves policy checks passed, while Terraform changes can be validated against organization policy before resources change.

Continuous delivery (CD) with Cloud Deploy is the automated progression of one versioned release through an ordered series of deployment targets. Cloud Deploy owns the delivery pipeline and promotion order, while Skaffold renders Kubernetes manifests and applies them, and Kustomize customizes shared manifests per environment. A release binds a rendering source and container images together, and each promotion creates a rollout to the next target.

2.1Two-Stage Rendering with Kustomize and Skaffold

Two-stage rendering in Cloud Deploy separates manifest generation from image injection. First, Kustomize generates the hydrated Kubernetes manifest from a kustomization.yaml file and its referenced resources. Then Skaffold substitutes image references in that generated manifest with the release images supplied at release creation, for example through --images=my-app-image=gcr.io/...@sha256:....

Cloud Deploy orchestrates this work through Cloud Build at release creation. The rendering source and image references are passed to Cloud Deploy, Cloud Build runs skaffold diagnose and skaffold render to produce a rendered manifest for every target profile before the first rollout, and Cloud Deploy later calls skaffold apply to apply the already-rendered manifest to each Google Kubernetes Engine (GKE) cluster. Because all targets are rendered up front, the full state can be reviewed, approved, and reapplied safely for rollbacks.

2.2Kustomize Paths and Skaffold Profiles

Kustomize configuration in skaffold.yaml tells Skaffold where to find the Kustomization to render. A Kustomize entry lives under the manifests stanza and points to directories containing a kustomization.yaml file, defaulting to ["."] when no path is set. Kustomize files can live locally or in Cloud Storage, and Artifact Registry can store artifacts such as Kustomizations.

Skaffold profiles are named variants inside one skaffold.yaml that select different rendering inputs for different targets. Each target that uses Kustomize gets its own profile with its own manifests.kustomize.paths value, such as environments/staging versus environments/prod. The delivery pipeline joins rendering to progression by listing profile names on each stage:

serialPipeline:
  stages:
  - targetId: staging-target
    profiles:
    - staging
  - targetId: prod-target
    profiles:
    - prod

When the profile name on the stage matches the profile name in skaffold.yaml, Cloud Deploy renders that stage with that stage's Kustomization. This separation lets teams change overlays without editing pipeline progression.

2.3Release Creation and Render-Apply Flow

The Cloud Deploy release flow carries CI output to cluster state through distinct Skaffold commands. Rendering happens for all targets at release creation, so approvers can inspect the exact YAML for each target before the first rollout.

1. Provide source and images: Pass the rendering source archive with skaffold.yaml plus --images mappings to Cloud Deploy.
2. Diagnose and render: Cloud Deploy calls Cloud Build, which runs skaffold diagnose to validate and skaffold render to hydrate manifests.
3. Substitute images: Skaffold replaces placeholder image names with tagged or digest-qualified release images.
4. Apply to first target: Cloud Deploy automatically creates a rollout that runs skaffold apply with kubectl as the deployer.
5. Promote: Each promotion applies the already-rendered manifest for the next target in sequence order.

Skaffold health checks run during deployment, and Cloud Deploy uses their results to determine rollout health. Cloud Deploy supports only kubectl as the deployer for standard targets, although Helm, Kustomize, and kpt can be used as renderers and custom targets can implement alternative logic.

2.4Strategies, Verification, and Deploy Hooks

Strategy is the per-stage pipeline setting that controls how a rollout places the new version onto a target. If strategy is omitted, Cloud Deploy uses the standard strategy, which deploys fully in one step.

Canary traffic splitting depends on the runtime. Cloud Run uses automatic traffic control, GKE service networking names an existing service and deployment, and GKE Gateway API names httpRoute, service, and deployment. Deployment verification runs immediately after the deploy job, while deploy analysis evaluates observability alerts over time. Predeploy hooks run before deploy and postdeploy hooks run after verify and analysis, so staging can use light checks while production uses canary phases and longer analysis.

Artifact Registry configuration is the design of format-specific repositories, access controls, lifecycle rules, and locations for container images and language packages. Each repository holds one format, such as Docker, Maven, npm, Python, Go, Helm, Apt, or Yum, and each repository uses a mode that controls whether it stores private artifacts, caches public artifacts, or forwards requests. Consumers such as Cloud Build, GKE, and Cloud Run benefit when retention and placement match their deployment pattern.

3.1Repository Modes and Dependency Flow

Repository modes in Artifact Registry (opens in a new tab) separate private storage, public caching, and consolidated access by how requests flow through them:

A standard repository depends on publishers such as Cloud Build to upload versions, while a remote repository depends on its external source on first request and then serves later requests from cache. A virtual repository depends on its upstreams, forwarding each client request in order until a match is found. Prioritizing the private standard repository above the remote repository reduces dependency confusion attacks, where an attacker publishes a higher-version public package with the same name.

3.2Security and Access Controls

Security in Artifact Registry combines identity, network, sourcing, scanning, and admission controls where artifacts are stored and requested. IAM permissions can be granted at the project level or narrowed to specific repositories, with build accounts receiving write access and runtime workloads receiving read-only access. Cross-project pulls require an explicit grant, such as granting the GKE node service account roles/artifactregistry.reader in the registry project.

Network isolation places a VPC Service Controls perimeter between trusted services inside the boundary, such as Cloud Build, GKE, and Cloud Run, and untrusted callers outside. Builds, pulls, and pushes inside the perimeter succeed, while outside requests are denied even with valid IAM credentials. An organization policy with a resource locations constraint further allows repository creation only in approved regions for data residency.

3.3Vulnerability Detection and Deployment Policy

Continuous vulnerability detection with Artifact Analysis (opens in a new tab) finds issues in stored artifacts through on-demand scans and automated scans on upload. The initial scan extracts system and language package inventory, and continuous monitoring keeps results current as new intelligence arrives by updating metadata, creating new occurrences, and deleting occurrences that are no longer valid. Teams can also generate an SBOM and upload VEX statements about exploitability.

Policy enforcement connects detection to provenance and admission. Cloud Build can produce SLSA Level 3 builds with non-falsifiable provenance recording digests, source locations, toolchain, steps, and duration. Binary Authorization then checks attestations before an image runs on GKE or Cloud Run, verifying the build system, vulnerability policy compliance, or promotion from development to quality assurance. Only images with required attestations deploy, so risky images remain in the registry.

3.4Lifecycle Policies and Multi-Regional Placement

Cleanup policies in Artifact Registry are named rules that automatically delete or preserve versions. Each policy has an id and an action of DELETE or KEEP, and policies can run in dry-run mode to audit impact before enforcement. Common conditions filter by tag state (tagged, untagged, any), tag prefix, package prefix, age since upload, and retained version count. A typical CI/CD pattern combines a DELETE rule for old development versions with KEEP rules for release prefixes or the newest N versions.

Repository location balances latency, availability, cost, and compliance. Placing the repository in the same region as GKE, Cloud Run, and Cloud Build reduces pull latency and avoids egress charges, since same-region egress to Google Cloud services has no charge. Multi-region to region egress is free only for limited mappings, such as us to a United States region, but charged to other continents. Image streaming for GKE also requires images in the same region or a corresponding multi-region.

Third-party tooling in Google Cloud CI/CD combines Git as the source of truth, an automation engine to build and test changes, and a delivery controller to reconcile environments. Cloud Build is the native engine that runs cloudbuild.yaml steps as containers, while Jenkins is a supported alternative for teams retaining existing investments. Cloud Deploy automates promotion to GKE and Cloud Run, while Argo CD acts as a GitOps controller that watches Git desired state and reconciles live clusters toward it.

4.1Git Events and Automation Engines

Git branch events provide the shared signals that start CI. Developers merge frequently into a shared repository, test automation runs unit and integration tests on each commit, and passing code produces versioned deployment artifacts. Both Cloud Build and Jenkins use build-time substitutions such as $COMMIT_SHA, $REPO_NAME, $BRANCH_NAME, and $BUILD_ID, and tagging images with $COMMIT_SHA links each image back to its exact commit for audit and rollback.

Cloud Build runs an ordered workflow against a repository that typically holds workflow source, Dockerfiles, tests, cloudbuild.yaml, and pipeline specifications:

1. Copy source: Copy the repository to /workspace in the build runtime.
2. Test: Run unit and integration tests, with optional static analysis.
3. Build images: Build one image per component and tag with $COMMIT_SHA if tests pass.
4. Publish images: Upload tagged images to Artifact Registry.
5. Rewire and compile: Update component specifications with new image URLs and compile the workflow to pipeline.json.
6. Publish and run: Upload pipeline.json and optionally run the pipeline for integration testing.

Triggers can be skipped for documentation-only edits to avoid unnecessary builds. Staged promotion then advances only verified artifacts, with preproduction validation required before production converges.

4.2Security, Identity, and Network Scope

Security for multi-cloud toolchains gives third-party tooling only the access it needs while proving inputs and outputs can be trusted. Least-privilege design uses unique service accounts per environment and project, separate accounts for reading versus writing artifacts, and separate accounts for managing Dataflow jobs versus running workers. Production jobs specify a user-managed worker with --serviceAccount instead of the broad Compute Engine default, tightened with IAM Conditions and just-in-time elevation. Secrets live in Secret Manager, and Identity-Aware Proxy adds context-aware access by IP and device.

Integrity controls address risks from Docker base images, operating system packages, and language libraries. Pipelines reduce substitution risk by requiring signed artifacts, maintaining trusted publisher keys, verifying signatures, and rebuilding infrastructure regularly from trusted sources. Network scope limits contain compromise by revoking unneeded write access, blocking internet egress or restricting it to an allow-list, and placing projects inside VPC Service Controls so deployed code cannot reach unauthorized APIs.

4.3Immutable Images with Packer and Declarative Config with kpt

Declarative configuration and immutable image creation defines infrastructure in versioned files and rebuilds images rather than patching systems. Packer automates golden machine images for Compute Engine, which are virtual disks that already contain required software, while Image Import brings existing VMDK or VHD disks into Google Cloud. kpt packages, customizes, and validates declarative Kubernetes configurations across GKE environments.

An automated image pipeline rebuilds on repository events through a custom image build pipeline (opens in a new tab) using cloudbuild.yaml and recipe files:

1. Trigger: A push to main after review starts the pipeline.
2. Validate and build: Cloud Build validates configuration and builds the custom OS image.
3. Publish: The pipeline publishes the validated image to Compute Engine and Artifact Registry.

This sequence enforces immutable infrastructure because a change produces a new image instead of modifying a running VM. For Kubernetes, Config Connector manages Google Cloud resources as Kubernetes objects, and Cluster Toolkit provides curated Terraform blueprints applied through Cloud Build. Tool choice follows responsibility: Terraform authors versioned configuration, Infrastructure Manager deploys it natively, and Terraform Cloud or Enterprise adds remote state and organizational change control.

Security of CI/CD tooling treats the pipeline itself like a production system with policy checks, gated promotions, and audited access. Cloud Logging records pipeline activity as an auditable trail, while Policy-as-Code engines check infrastructure as code before and after deployment. Prevention at check-in blocks misconfigurations early, and detection in production catches drift when policies or workloads change later.

5.1Policy-as-Code and Audit Logging

Policy-as-Code writes compliance rules as machine-readable files and checks infrastructure automatically in the pipeline. Open Policy Agent is the general policy engine, and Policy Controller is its Kubernetes admission enforcement point that blocks noncompliant resources. Central Google Cloud guardrails surround the pipeline infrastructure: IAM authorizes who can act, Organization Policy restricts how resources can be configured, and VPC Service Controls creates service perimeters for defense in depth.

Organization policies commonly applied to CI/CD environments include:

• Limit resource sharing based on domain to prevent cross-organization access.
• Limit the use of service accounts to control which identities pipelines can use.
• Restrict the physical location of newly created resources to meet residency requirements.

Cloud Deploy adds human oversight through gated deployments that require explicit approval before promotion to development, test, or production targets. Pipeline health is tracked with dashboards, alerting, and Service Level Indicators and Objectives for build and deployment latency, while breakglass procedures require a clear escalation plan, shared custody of secrets, and automated auditing of every emergency use.

5.2Binary Authorization and Continuous Validation

Binary Authorization is the pre-deployment gate that allows only trusted images to deploy to GKE, Cloud Run, and GKE Enterprise. The policy maintains an exempt image list and can require attestations, which are signed documents stating that a required process completed, such as a Cloud Build origin or a vulnerability severity check. A missing or invalid attestation stops promotion and preserves supply-chain integrity.

Continuous validation extends that policy to already-running Pods throughout their lifecycle. It does not block running Pods but regularly writes conformance results to Cloud Logging, including Pods admitted through dry-run mode or breakglass bypass. The GKE security posture dashboard adds fleet-level scanning against best practices such as Pod Security Standards, with severity ratings and remediation guidance also logged for audit.

5.3Supply Chain Integrity and Least-Privilege Isolation

Software supply chain integrity connects verifiable builds, centralized storage, continuous scanning, and deployment control into one ordered flow. Cloud Build supports SLSA Level 3 builds and generates authenticated provenance linking image digests to source locations, toolchain, steps, and duration. Artifact Registry centralizes artifacts with remote repositories that cache external sources and virtual repositories that prioritize private packages, while Artifact Analysis scans uploads, monitors for new findings, and supports SBOM and VEX. Binary Authorization then admits only images with required builder, vulnerability, and promotion attestations.

Secure execution means a compromised step cannot read, write, or exfiltrate beyond its need. Each pipeline identity receives only minimum permissions scoped to individual resources, with dedicated accounts per pipeline or environment and Secret Manager for credentials. VPC Service Controls places a perimeter around projects and APIs so code inside cannot call outside resources. Two exfiltration paths drive design: direct exfiltration copies data through pipeline access and is blocked by revoking unneeded access and egress, while indirect exfiltration deploys compromised code that steals data and is contained by the perimeter.

Key Facts

• A cloudbuild.yaml file defines ordered builder steps that share a workspace, and a Cloud Build trigger binds source, branch filter, config path, substitutions, and service account so only matching pushes build.
• Tagging images with $SHORT_SHA or $COMMIT_SHA ties each Artifact Registry image to its exact commit, and that tag is written into Kubernetes manifests to link integration output to delivery.
• Cloud Deploy renders manifests for every target at release creation through Cloud Build running skaffold diagnose and skaffold render, then runs skaffold apply with kubectl on each rollout, with Skaffold profiles selecting per-target Kustomize overlays.
• Artifact Registry uses standard repositories for private artifacts, remote repositories to cache public dependencies, and virtual repositories to serve both through one endpoint with private packages prioritized to reduce dependency confusion.
• Binary Authorization blocks deployment unless required attestations are present, while Artifact Analysis detects vulnerabilities and continuous validation only logs violations for already-running Pods.