Pipeline Secrets with KMS and Secret Manager

Envelope encryption for CI/CD pipelines is a layered design where a per-object data encryption key (DEK) encrypts the payload and a customer-managed key encryption key (KEK) wraps that DEK. Cloud Key Management Service (Cloud KMS) (opens in a new tab) holds the KEK and never exposes raw KEK material, while the wrapped DEK travels and is stored alongside the encrypted payload in Secret Manager, Cloud Storage, or another CMEK-integrated service. Reading the payload therefore requires both permission on the resource and permission to unwrap the DEK with the KEK.

1.1Envelope Encryption Flow and Customer-Managed Keys

Envelope encryption with customer-managed encryption keys (CMEK) separates the key that encrypts data from the key that encrypts other keys. The flow starts when a CMEK-integrated service generates a DEK, uses the DEK to encrypt the secret value or artifact with AES-256, then sends the DEK to Cloud KMS where the CMEK encrypts the DEK as KEK. For decryption, the service sends the wrapped DEK back to Cloud KMS, Cloud KMS decrypts the DEK with the CMEK, and the service uses the unwrapped DEK in memory to decrypt the payload.

All CMEK keys use the symmetric ENCRYPT_DECRYPT purpose with the GOOGLE_SYMMETRIC_ENCRYPTION algorithm, and each key supports unlimited key versions. The primary version encrypts new DEKs, while older enabled versions remain available to decrypt previously wrapped DEKs. If the CMEK is disabled, destroyed, or its roles/cloudkms.cryptoKeyEncrypterDecrypter grant to the service agent is revoked, decrypt operations fail and protected secrets, buckets, and build outputs become unreadable.

The choice between Google-managed and customer-managed keys determines ownership, visibility, and control:

1.2Protecting Secrets, Buckets, and Build Artifacts

Secret Manager payload protection with CMEK applies envelope encryption to secret values while preserving transparent access for authorized pipelines. After CMEK is configured, a build step that has Secret Manager access does not need direct access to the KEK, because the Secret Manager service agent uses its Encrypter/Decrypter grant to unwrap the DEK and return plaintext. This trust boundary separates secret consumers in the application or pipeline project from key administrators in the key project.

Replication and location determine where wrapped secrets and KEK operations can occur. Automatic replication is recommended for most workloads because it meets availability and performance needs without pinning replicas. User-managed replication through the API is used only for strict residency requirements, and the Cloud KMS key ring must reside in a location compatible with those replicas. The KEK should be created in the same location family as the resources it protects to keep encrypt and decrypt calls local and policy-compliant.

Cloud Storage bucket protection with CMEK applies the same pattern to objects that hold build artifacts, image layers, and deployment bundles. When an object is written, Cloud Storage generates a DEK for the object, wraps that DEK with the bucket CMEK, and stores the wrapped DEK with object metadata. On read, Cloud Storage calls Cloud KMS to unwrap the DEK and then decrypts the object, so pipeline identities need object access while only the Cloud Storage service agent needs the KEK permission. A key ring for a bucket must be in the same location scope as the bucket, such as the US multi-region for a bucket spanning us-east1 and us-west1. Objects encrypted with CMEKs do not return CRC32C or MD5 hashes on JSON API listings, so inventory tools that fetch checksums per object list substantially slower.

1.3Manual Provisioning, Autokey, and Governance Models

Key provisioning for pipelines follows either manual creation or Cloud KMS Autokey, and both produce standard Cloud Hardware Security Module (HSM) keys that behave identically in CMEK services. Autokey generates key rings, AES-256 GCM keys, and service-agent grants on demand as developers create resources, with multi-tenant HSM protection, yearly rotation, same-location placement, and per-resource-type specificity. Manual provisioning requires advance planning of rings, keys, locations, and Identity and Access Management (IAM) grants, but allows full control over naming, granularity, and protection level.

Manual CMEK setup for a pipeline resource follows an ordered dependency chain where each step depends on the prior step:

1. Create or select a key ring: Create the key ring in a location near the secrets, buckets, or build resources it will protect.
2. Create or import the key: Create the symmetric encryption key in that key ring; this key becomes the CMEK KEK.
3. Grant the service agent access: Grant roles/cloudkms.cryptoKeyEncrypterDecrypter on that CMEK to the service agent of Secret Manager, Cloud Storage, Cloud Run, or the relevant build service.
4. Configure the resource to use the CMEK: Attach the CMEK at resource creation so all subsequent writes are envelope-encrypted.

Governance and storage models determine who manages keys and where keys live. Centralized governance places lifecycle responsibility with a security or platform team and suits regulated environments, while delegated governance lets application owners manage keys within guardrails such as organization policies and IAM deny policies. Dedicated-project storage places keys for multiple applications in a centralized key project per environment, separate from protected resources, while same-project storage places the key with the data it protects. Use one centralized key project per environment and do not co-locate keys with protected resources when separation of duties is required.

Protection level and key material choices trade cost, control, and overhead:

• Choose Cloud External Key Manager (EKM) when key material must remain outside Google Cloud.
• Choose single-tenant Cloud HSM when key material must reside on dedicated partitions of Google-owned HSMs.
• Choose multi-tenant Cloud HSM when shared Google-owned HSM clusters are acceptable, which is also the Autokey default.
• Choose software protection only when HSM or external requirements do not apply.

Prefer Google-generated key material because it never exposes raw material and supports automatic rotation. Imported bring-your-own-key material requires external generation, secure escrow, leak-resistant import automation, and planning for failed rotations or re-import of destroyed versions.

1.4Rotation, Destruction Lifecycle, and Detective Controls

Key rotation in Cloud KMS creates a new primary version for new encrypt operations while older enabled versions remain required for decrypt, which lets pipelines keep running. Automatic rotation should be enabled at least once per year, and more often for sensitive workloads. Rotation behavior differs by service: Spanner re-encrypts the DEK with the new version, Cloud Storage applies the new version only to newly written objects, and BigQuery by default keeps existing tables on the old version until the table is recreated or explicitly re-encrypted. Old versions must stay enabled until dependent data is re-encrypted, migrated, or purged.

The destruction lifecycle moves a version through ordered states that control pipeline access. A version starts enabled for encrypt and decrypt, can move to disabled to block all use without deleting material, then to scheduled for destruction where all cryptographic operations fail, and finally to destroyed where material is irrecoverably deleted. The scheduled for destruction duration can only be set at creation, defaults to 30 days, and can be set as low as 24 hours. Preventive guardrails include constraints/cloudkms.minimumDestroyScheduledDuration with at least 30 days, constraints/cloudkms.allowedProtectionLevels to limit backends, constraints/gcp.restrictNonCmekServices to require CMEK, and project liens to block accidental key-project deletion. Detective controls aggregate Admin Activity logs centrally, enable Data Access logs for encrypt and decrypt only with a clear use case because volume increases cost, and use the inventory API, key-usage dashboard, Cloud Monitoring alerts on destruction scheduling, and Security Command Center findings for overly permissive or publicly accessible keys.

Secret and parameter lifecycle engineering is the practice of storing sensitive configuration in managed stores and controlling replication, encryption, access, and audit from pipeline to runtime. Secret Manager (opens in a new tab) is the IAM-controlled store for secrets, while Cloud KMS provides customer-controlled keys that protect data at rest. Centralized projects for each service let a dedicated team own creation, rotation, and auditing to meet compliance requirements.

2.1Secret Storage, Replication, and Encryption Lifecycle

Secure storage in Secret Manager depends on a dedicated centralized project that hosts secrets for pipelines and workloads. A workload identity calls the Secret Manager API, and Secret Manager grants or denies the read based on IAM permissions. This separation means pipeline execution and application runtime depend on Secret Manager for secret material, while ownership and audit remain with the central security team.

Replication policy in Secret Manager controls where payloads are stored and served from, which affects availability, performance, and residency:

Encryption lifecycle starts with Google default encryption at rest and in transit, with CMEKs added when regulatory or rotation control is required. Cloud KMS holds CMEKs as software-based keys or HSM-protected keys, with Autokey automating provisioning and Cloud EKM supporting key material that stays in a third-party system outside Google Cloud. A centralized Cloud KMS project lets a dedicated team manage keys separately from application teams. Key rings must be in a location compatible with the data, so data in us-east1 and us-west1 requires a US multi-region ring, and predefined dual-regions such as asia1, eur4, and nam4 require a ring in that same dual-region. When a workload reads a secret, data flows from the storage service to Cloud KMS for decryption, so loss of access to the key blocks access to the data.

2.2Least-Privilege Access and Governance Guardrails

Least-privilege access for secrets means every pipeline step and runtime workload receives only the permissions needed for its task. CI/CD service accounts receive only the secret accessor roles they require, and Workload Identity Federation lets applications on Google Kubernetes Engine (GKE) or outside Google Cloud access Secret Manager without long-lived keys. Audit logging, permission review, and IAM Recommender suggestions further reduce excess privilege. Defense in depth extends beyond IAM to the network path: VPC Service Controls creates perimeters around Secret Manager and Cloud KMS to help prevent exfiltration, while strict egress rules and private connectivity keep secret reads off the public internet.

Organization Policy Service enforces centralized guardrails across organization, folder, and project levels. The constraint constraints/gcp.restrictNonCmekServices requires CMEK protection for listed services such as bigquery.googleapis.com, storage.googleapis.com, and aiplatform.googleapis.com. When this policy applies, resource creation without a customer-managed key is denied, so pipelines cannot deploy non-compliant storage. Complementary constraints restrict resource locations, limit service account creation, and disable external IP allocation.

2.3Keyless Authentication with Workload Identity Federation

Workload Identity Federation is a keyless authentication method that lets workloads outside Google Cloud access Google Cloud APIs without long-lived service account keys. The external workload presents a credential from its own external identity provider (IdP), and Google Cloud exchanges that credential for short-lived credentials. This exchange removes the need to download, store, and rotate RSA service account keys in pipeline configuration.

Service account keys are long-lived private keys that can authenticate from anywhere and create risk if leaked or over-shared. Federation replaces them with short-lived credentials issued on demand. The workload obtains an environment-native credential, exchanges it with the Security Token Service (STS) for a federated token, and then calls Google Cloud APIs with OAuth 2.0 access tokens that expire by default after 1 hour. Supported external environments include:

• AWS EC2 instances using instance profiles and temporary credentials
• Azure VMs using managed identities and Azure access tokens
• Kubernetes clusters on AKS, EKS, or self-hosted clusters using projected ServiceAccount tokens
• External CI/CD platforms such as GitHub that issue OpenID Connect (OIDC) tokens

A workload identity pool is the IAM trust boundary that represents external identities, and best practice manages pools and providers in a dedicated project separate from applications. A workload identity pool provider describes a single external IdP with its issuer URL, expected audience, and verification method such as OIDC discovery and JSON Web Key Set (JWKS). Attribute mapping translates external claims such as repository, branch, AWS role, Azure principal, or Kubernetes namespace into federated attributes including google.subject, while an attribute condition restricts which external identities can federate.

2.4Token Exchange Flow and Authorization Models

Token exchange for external workloads is the ordered sequence where each credential depends on the previous credential. Traffic flows from the external workload to STS and then to the target Google Cloud API, while trust flows from the external IdP to the pool provider to IAM allow policies. Client libraries, gcloud CLI, and Terraform perform the exchange automatically when pointed at a credential configuration file.

1. Obtain external credential: The workload obtains its native credential, such as a GitHub OIDC token, AWS temporary credential, Azure access token, or Kubernetes projected token.
2. Exchange with STS: The workload sends the external credential to STS referencing the pool and provider resource name. STS validates it against the provider, applies mapping and conditions, and returns a short-lived federated token.
3. Access Google Cloud resources: The federated token is used directly or to impersonate a service account, and the workload calls the target API with short-lived credentials.

Direct resource access and service account impersonation differ in which principal receives the role:

Direct access is recommended when the API supports federated identities because it avoids an extra service account. Impersonation is required when an API only accepts a service account or when many identities should share one audited identity, and it requires granting the federated principal roles/iam.workloadIdentityUser on the service account. A credential configuration file generated with gcloud iam workload-identity-pools create-cred-config records the provider, credential source, and optional impersonated account, and tools discover it through GOOGLE_APPLICATION_CREDENTIALS. For GKE, Google manages the pool automatically with the fixed name PROJECT_ID.svc.id.goog.

2.5Automated Certificate Provisioning and Mapping

Automated certificate provisioning with Certificate Manager (2nd gen) issues, attaches, selects, and renews TLS certificates for load balancers and endpoints without manual replacement. Certificate Manager centralizes the lifecycle across load balancers, GKE, Compute Engine, and hybrid environments through issuance configurations, certificate maps, and integration with Certificate Authority Service and Public CA. Google-managed certificates renew automatically once domain ownership is proven, while self-managed certificates require orchestrated rotation.

Certificate maps bind certificates to a load balancer target proxy and control selection during the TLS handshake. A map supports thousands of entries by default and can handle millions for multi-domain deployments. When a map is used, map entries take precedence over Compute Engine SSL certificates on the same proxy. If a client hostname matches no entry, the load balancer serves the designated primary certificate as fallback.

Domain authorization proves ownership before issuance, and the method determines speed, network design, and wildcard support:

Issuance configurations automate generation and rotation by linking a workload identity pool to a Certificate Authority Service pool, controlling lifetime, key algorithm, and rotation window. The same mechanism supports mutual TLS (mTLS) between Application Load Balancers and backends and managed workload identity for Compute Engine and GKE. Self-managed certificates from an external or private CA follow an ordered rotation: monitor expiration with Cloud Monitoring alerts for 15 to 30 days, generate a signing request and private key, obtain the new certificate, then upload it and swap the map entry reference in one API call so new handshakes move with no gap.

GCP-specific secret management separates build-time provisioning from runtime injection by where the secret resolves and how it reaches the workload. Secret Manager (opens in a new tab) is the central store for both phases, Cloud Build consumes secrets during the build, and services on GKE, Cloud Run, or Vertex AI Pipelines fetch secrets after deployment. Build-time provisioning makes a secret available to an ephemeral build step, while runtime injection makes the live value available to a running workload.

3.1Build-Time Provisioning with Cloud Build

Build-time provisioning with Cloud Build and Secret Manager delivers a secret version to specific build steps without storing the value in source control. The flow runs from Secret Manager to the build step to the consuming process: the secret and version are stored in Secret Manager, the build configuration maps that version to a local variable name, and only steps that declare access receive the value in memory. UTF-8 secrets use the environment-variable pattern, while non-UTF-8 secrets require a separate binary-safe path.

Configuration uses the availableSecrets block after all build steps in cloudbuild.yaml or cloudbuild.json in this order:

1. Declare the mapping: Add availableSecrets.secretManager with pairs of versionName, such as projects/PROJECT/secrets/SECRET/versions/VERSION, and env, the local variable name.
2. Scope access per step: In each step that needs the secret, set entrypoint to bash, add secretEnv listing allowed env names, and start args with -c.
3. Consume without persisting: Use the variable inside scripts, Docker authentication, or API calls during that step, so the value resolves at build time and is not committed.

The build service account must hold Secret Manager Secret Accessor (roles/secretmanager.secretAccessor) on the specific secret before the build runs, and both the Cloud Build and Secret Manager APIs must be enabled.

3.2Runtime Injection for GKE, Cloud Run, and Vertex AI

Runtime injection for deployed workloads keeps secrets out of deployment artifacts and Cloud Deploy pipelines. The running Pod, Cloud Run service, or Vertex AI task authenticates to the Secret Manager API and reads a specific secret version, so secrets are generated, managed, and rotated outside the delivery pipeline. Provisioning creates or updates the secret before the application deploys, so the workload never starts without its required version.

The preferred pattern is direct API access from application code using client libraries, REST, or gRPC with Application Default Credentials. Direct access avoids expanding the trust boundary to the filesystem or process environment, where traversal flaws can read files and debug endpoints can leak variables. Platform integrations still support environment variables or volume mounts when direct calls are impractical, including native Cloud Run secret bindings and the Secrets Store CSI driver that mounts secrets for GKE with Workload Identity authentication. Non-sensitive metadata such as version number or environment name can be passed as ordinary environment variables while secret material stays in Secret Manager.

Authentication uses platform credentials instead of exported keys, with gcloud auth application-default login locally, the metadata server on Compute Engine and Cloud Run, Workload Identity on GKE, and workload identity federation on AWS or Azure. Project, secret, and network boundaries prevent one compromise from crossing environments by segmenting staging and production into separate projects, preferring secret-level bindings or IAM Conditions, using IAM Recommender to find excess privilege, and adding VPC Service Controls perimeters, constraints/iam.allowedPolicyMemberDomains limits, and AccessSecretVersion Data Access logs at folder or organization level.

3.3Security and Operational Tradeoffs of Injection Methods

Security posture differs because build-time embedding couples the secret lifecycle to the artifact lifecycle, while runtime fetching decouples them. Build-time embedding bakes values into images, manifests, .env files, layer history, build logs, and Software Bill of Materials (SBOM) metadata, so anyone who can read the artifact can recover the secret. Cloud Deploy pipelines should not manage application secrets this way because injecting secrets into deployment artifacts introduces security risks. Build-time values also break least privilege because artifact readers, scanners, and approvers gain indirect access, and rotation requires rebuilding, re-scanning with Artifact Analysis, re-attesting with Binary Authorization, and redeploying.

Runtime injection is stronger because the secret stays in Secret Manager and the workload fetches the decrypted value at execution with per-secret accessor grants and a full audit trail. High-value payloads can be wrapped with CMEK in Cloud KMS, verified with integrity checksums, replicated automatically or stored as regional secrets for residency, and isolated with VPC perimeters. Automated rotation publishes a SECRET_ROTATE notification to Pub/Sub when the interval starts, then a Cloud Run function or service generates a new value, adds the new version, and destroys the retired version.

Operational consistency depends on lifecycle discipline. References should pin a version number rather than the latest alias, so configuration validates through release processes and rolls back by redeploying the prior number. Versions move through ordered states: first disable a version to reversibly block use, confirm no lingering build or runtime dependencies, and only then destroy or delete. Do not set expiration on production secrets unless irreversible deletion is intended. Capacity planning must estimate peak AccessSecretVersion usage because concurrent deploys or autoscaling can create bursts that exceed quota.

3.4Pipeline Architecture for Secure Lifecycle Management

Secure lifecycle management designs CI/CD pipelines so secrets are created, stored, rotated, and revoked outside the deployment system and fetched only when needed. Cloud Build builds artifacts and Cloud Deploy delivers them to GKE or Cloud Run without embedding values, while the pipeline depends on Secret Manager for values, IAM for checks, and Cloud Logging and Pub/Sub for audit and rotation events.

Segregated access means every stage and microservice can access only its required secrets. Architecture starts by segmenting environments into separate projects for isolated IAM and quotas, then narrows access to individual secrets rather than whole projects. Practical controls include secret-level accessor bindings, IAM Conditions such as allowing triggers only from trusted build triggers, tags for role mapping, principal access boundary policies and Privileged Access Manager for just-in-time elevation, and keyless authentication. Confidentiality follows the Bell-LaPadula direction where data flows from lower to equal confidentiality only, while integrity follows the Biba direction where pipelines do not consume lower-integrity inputs for higher-integrity outputs.

Isolated architecture limits blast radius by keeping each pipeline scope small, avoiding cross-project access, and treating CI/CD tooling with production integrity through verified tools, rebuilt infrastructure, and shielded virtual machines (VMs). Exfiltration can be direct, where the pipeline identity reads confidential data and copies it out, or indirect, where the pipeline creates compute with attached service accounts and malicious code steals data from inside Google Cloud. VPC Service Controls contains the indirect path by placing protected resources inside a perimeter with the pipeline outside, so traffic flows from the pipeline to the perimeter only through ingress rules limited to pipeline addresses and needed services. Continuous monitoring correlates Admin Activity, Data Access, and pipeline logs, routes logs to long-term storage when retention misaligns, enriches attribution with run IDs, and uses SLSA Level 3 builds and Binary Authorization to promote only attested images.

Key Facts

• Envelope encryption uses a per-object DEK to encrypt data and a customer-owned KEK in Cloud KMS to wrap the DEK, with all CMEK keys using symmetric ENCRYPT_DECRYPT with GOOGLE_SYMMETRIC_ENCRYPTION.
• A key version moves from enabled to disabled to scheduled for destruction to destroyed, with a default 30-day soft-delete period and a minimum of 24 hours settable only at creation.
• Workload Identity Federation exchanges an external credential via Security Token Service for short-lived credentials, with OAuth 2.0 access tokens expiring by default after 1 hour.
• Cloud Build provisions build-time secrets through an availableSecrets.secretManager mapping of versionName to env combined with per-step secretEnv, while runtime workloads fetch secrets via direct API calls, Cloud Run bindings, or Secrets Store CSI driver.
• Only DNS authorization in Certificate Manager supports wildcard certificates, and certificate map entries take precedence over Compute Engine SSL certificates on the same target proxy.