Data Encryption with Cloud KMS and TLS

Customer-managed and customer-supplied keys are two ways to control the encryption keys that protect customer content stored at rest in Google Cloud. Google Cloud encrypts stored data by default, while customer-managed encryption keys (CMEKs) let you own keys in Cloud Key Management Service (Cloud KMS) (opens in a new tab) and customer-supplied encryption keys (CSEK) let you supply raw key material with a request. The choice determines who controls key lifecycle, auditability, and where key material can reside, which matters for compliance and data sovereignty.

1.1Key Management Models for Data at Rest

The key management model defines who generates the key, where the key lives, and who can revoke access to the data. Default encryption (opens in a new tab) uses Google-owned and Google-managed keys at no extra cost and covers stored customer data without customer action. CMEK moves key ownership to the customer while Google services still perform encryption, and CSEK moves key supply to the customer on a per-operation basis.

CMEK protection depends on access to the Cloud KMS key, so disabling, destroying, or revoking access to the key makes the protected data undecryptable even though the encrypted bytes remain in storage.

1.2How CMEK Uses Envelope Encryption with Cloud KMS

Cloud KMS is the service for creating, storing, rotating, and using cryptographic keys for CMEK integrations. Cloud KMS supports generating software keys, generating hardware keys in Cloud HSM, importing existing keys, and linking external keys held in a compatible external key management system. CMEK integrations use envelope encryption, where the Cloud KMS key acts as a key-encryption key (KEK) that wraps the per-chunk data-encryption keys (DEKs) that directly encrypt storage chunks with AES-256.

When a supported service reads or writes CMEK-protected data, the service calls from its service account to Cloud KMS to unwrap the DEK and then uses the unwrapped DEK to encrypt or decrypt the chunk. This flow creates a central chokepoint because data flows from the storage service to Cloud KMS for key use and back to the storage service for data access, and Cloud KMS audit logs record each use. Cloud KMS Autokey (opens in a new tab) automates provisioning by generating key rings, keys, and service accounts on demand as part of resource creation, which reduces the need to pre-create keys for every pipeline project.

1.3Applying CMEK to Data Pipeline Assets

A CMEK integration lets a Google Cloud service encrypt its data at rest with a Cloud KMS key that you own and manage. Data protected this way cannot be decrypted without granting the service identity permission to use that specific key version, which gives granular control per dataset, bucket, topic, or cluster and supports separation by environment or data domain.

Relevant pipeline assets with CMEK support include:

• BigQuery datasets and tables for warehouse storage
• Cloud Storage buckets for lake, staging, and backup objects
• Pub/Sub topics for message data associated with topics
• Dataflow pipeline state data for streaming and batch execution
• Dataproc clusters and serverless VM disks, Dataproc Metastore, and Dataform repositories for processing metadata
• Cloud SQL, Spanner, Firestore, and Bigtable for transactional and analytical stores
• Compute Engine persistent disks, snapshots, custom images, and machine images for VM-based workers

A CMEK-compliant service is different from a CMEK integration because the compliant service does not store data long term. Such a service encrypts short-lived data with an ephemeral key that exists only in memory and is never written to disk, and the ephemeral key is flushed from memory when processing ends so the data cannot be recovered. Pipeline outputs from a compliant service can then be persisted in a CMEK-integrated service such as Cloud Storage or BigQuery to retain CMEK control.

1.4Enforcing Lifecycle, Residency, and Compliance with CMEK

CMEK lifecycle control means you manage creation, rotation, version status, destruction, and audit of the keys that protect pipeline data. Keys can be rotated on a schedule, disabled to suspend access, and have versions destroyed or deleted to make data permanently unreadable, which supports secure disposal and incident response. Protection level choices add assurance, including FIPS 140-2 Level 3 certified hardware for key generation in Cloud HSM and storage of key material outside Google Cloud with Cloud EKM for sovereignty requirements.

Central enforcement depends on organization policy rather than on each developer remembering to attach a key. The security team can block creation of non-compliant resources with CMEK organization policies, require CMEK on buckets, datasets, tables, disks, topics, and clusters, and restrict non-CMEK services in regulated projects. This division places key governance with the security team while application and data developers select the approved key when they create each pipeline asset.

1.5Role of CSEK and Cloud EKM

CSEK is a model where the customer provides the key material directly for a specific resource operation, historically used for Compute Engine persistent disks. CSEK coverage is narrow, places per-request key handling on the caller, and does not provide the centralized lifecycle, rotation, policy enforcement, and broad service support of Cloud KMS. For that reason, scenarios historically addressed by CSEK are now better addressed by Cloud EKM, which supports more services and higher availability while still keeping key material outside Google Cloud.

Implementing and managing encryption keys means using Cloud Key Management Service (Cloud KMS) (opens in a new tab) to create and control customer-managed encryption keys (CMEKs) that protect data pipelines in services such as Cloud Storage, BigQuery, and Compute Engine. A CMEK design covers where keys live, who manages them, what hardware protects them, how often they rotate, and how their use is guarded and monitored. When these choices align, each dataset gets a cryptographic boundary that the data owner controls from creation through destruction.

2.1Ownership, Envelope Encryption, and Default Encryption

A customer-managed encryption key is a Cloud KMS key that the customer owns and controls to protect resources in CMEK-integrated services. By default, data at rest in Google Cloud is protected by Google-owned and Google-managed keys in Keystore, where Google handles rotation automatically, data from multiple customers can share the same key-encryption key, and key metadata and usage logs cannot be viewed. With CMEK, Cloud KMS uses envelope encryption (opens in a new tab) in which the CMEK acts as a key-encryption key (KEK) that encrypts the data-encryption keys (DEKs) used by Cloud Storage, BigQuery, Compute Engine, and other services.

CMEK ownership enables control over location, protection level, creation, access, rotation, use, and destruction, plus selective deletion of protected data through crypto-shredding by disabling or destroying the key. CMEK keys are unique to a customer, which establishes logical data separation so owners do not share KEKs and DEKs across trust boundaries. All CMEK keys for data-at-rest protection must use the symmetric ENCRYPT_DECRYPT purpose with the GOOGLE_SYMMETRIC_ENCRYPTION algorithm, and workloads access CMEK-protected resources the same way as with default encryption.

2.2Key Rings, Projects, and Governance Models

A key ring is a Cloud KMS container that groups keys in a specific project and location, and keys can only encrypt resources according to location and policy constraints. The recommended pattern is one centralized key project for each environment, with a key ring in each location where protected Cloud Storage buckets, BigQuery datasets, or Compute Engine disks reside. Cloud KMS resources should not be created in the same project as the data they protect, because mixing keys and data makes separation of duties between key administrators and data administrators harder to enforce.

Key governance describes who manages the key lifecycle and guardrails, while key storage describes where Cloud KMS resources are created. Centralized governance assigns all key lifecycle work to a dedicated security or platform team and is preferred for highly regulated workloads or external key systems, while delegated governance lets a central team set organization policies and Identity and Access Management (IAM) guardrails and lets application owners manage keys in their own projects to avoid bottlenecks. For storage, dedicated-project key storage places keys for multiple applications in a dedicated key project per environment folder, while same-project key storage places the key in the same project as the data it protects so the key follows the data. Both storage models work with Autokey, but dedicated-project storage better supports centralized control, least privilege with limited predefined roles, and removal of broad owner roles.

2.3Protection Levels and Sources of Key Material

The protection level determines where key material lives and what hardware or external system protects it. Cloud KMS supports software, hardware, and external backends, and the choice controls compliance, performance, cost, and operational responsibility.

When a key is created, key material is either generated inside Cloud KMS or imported from outside Google Cloud as bring-your-own-key (BYOK). Google-generated material is preferred because raw material is never exposed outside Cloud KMS and new versions can be created automatically on the rotation schedule. Imported material requires automation to generate and import new versions on time, secure storage or escrow of the original material, controls to prevent leakage during import, and planning for the risk that retained outside material could be used to re-import a destroyed key.

2.4Rotation Policies and Service-Specific Behavior

Automatic rotation creates a new primary key version on a schedule, while older versions remain available to decrypt existing DEKs. The recommended practice is to enable automatic rotation with a period that meets compliance needs, ideally at least once per year, with more frequent rotation for sensitive pipelines. Rotation limits the amount of data protected by any single version and makes scheduled replacement predictable for auditors and operators.

Rotation of the Cloud KMS key does not re-encrypt all existing data in the same way, because each integrated service applies the new version according to its own envelope-encryption design. In Spanner, a new key version re-encrypts the DEK that protects the data. In Cloud Storage, only newly written objects use the new key version, so different versions of the same key protect different subsets of bucket data unless existing objects are rewritten. In BigQuery, an existing table continues to use the key version assigned at creation time and does not automatically switch when the Cloud KMS key rotates, so the table must be recreated or reconfigured to adopt the new version.

2.5Cloud KMS Autokey for On-Demand Provisioning

Cloud KMS Autokey is the automated provisioning mode in which a Cloud KMS administrator delegates key-creation authority to the Cloud KMS service agent, and developers request keys on demand when they create protected resources. Manual provisioning requires administrators to plan key rings, keys, and service accounts in advance and coordinate with developers, while Autokey creates and assigns them at request time with consistent settings that respect separation of duties. Autokey is available in all Google Cloud locations where Cloud HSM is available and simplifies infrastructure-as-code because Terraform no longer needs elevated key-creation privileges.

Keys created by Autokey follow industry-standard defaults for pipeline data protection:

• HSM protection level keys stored in the Cloud HSM backend as AES-256 GCM encryption keys.
• Yearly automatic rotation with customer control over schedule, IAM, and enable, disable, or destroy actions.
• Co-location of each key in the same location as the resource the key protects.
• Per-resource-type specificity so one key does not over-share across unrelated resource types.
• Separation of duties so identities that encrypt and decrypt are different from identities that manage rotation, destruction, or state changes.

2.6Destruction Lifecycle, Guardrails, and Detective Controls

The key-version lifecycle moves through enabled, disabled, scheduled for destruction, and destroyed, with restore possible only during the soft-delete window. The default scheduled-for-destruction period is 30 days, but a key can be created with a period as short as 24 hours, which may leave too little time to detect accidental destruction. The destruction duration can only be set during key creation, and while a key is scheduled for destruction it cannot perform cryptographic operations, so requests to use it fail until the key is restored or the period expires and the material is irrecoverably destroyed.

Preventive guardrails enforce these lifecycle choices through organization policy constraints and project controls. The constraints/cloudkms.minimumDestroyScheduledDuration constraint requires a minimum destruction window such as 30 days, constraints/cloudkms.allowedProtectionLevels limits new keys, versions, and import jobs to approved protection levels, and constraints/gcp.restrictNonCmekServices requires CMEK protection where mandated. Project liens prevent accidental deletion of centralized key projects, while least privilege and separation of duties keep key administrators distinct from principals that use keys for encryption and decryption.

Detective controls reveal how keys are configured and used across Cloud Storage, BigQuery, and Compute Engine. Centralized Admin Activity audit logs record creation and modification of Cloud KMS resources for the whole organization, while Data Access logs for encrypt and decrypt operations are optional because every CMEK operation can create large log volume and cost. The Cloud KMS inventory API and key-usage dashboard identify which resources depend on each key and flag data made inaccessible by a disabled or destroyed key, Cloud Monitoring can alert on sensitive events such as scheduling destruction, and Security Command Center (opens in a new tab) surfaces misconfigurations such as publicly accessible keys, overly permissive owner roles, or separation-of-duties violations.

Encryption for data in transit in Google Cloud protects data while it moves between users, services, and environments. Google Cloud (opens in a new tab) encrypts and authenticates traffic by default when it leaves physical boundaries controlled by Google, and it provides additional controls for application traffic, private service perimeters, and hybrid links. Planning a pipeline means deciding when default protection is enough and when to add Transport Layer Security (TLS), private service perimeters, or link-layer encryption.

3.1Google Cloud Default Protection Inside Google Networks

Google Cloud default encryption in transit covers common internal traffic patterns without customer configuration. Traffic between virtual machines (VMs) within a Virtual Private Cloud (VPC) network and between peered VPC networks inside Google's production network is encrypted, integrity-protected, and authenticated. Traffic from the internet to Google services and from a VPC network to Google services terminates at the Google Front End (GFE), which provides distributed denial-of-service (DDoS) countermeasures and routes and load-balances traffic to the service.

From the GFE onward, Google's infrastructure uses Application Layer Transport Security (ALTS) for authentication, integrity, and encryption of connections from the GFE to a Google Cloud service and from one Google Cloud service to another Google Cloud service. This means a call from a pipeline worker to BigQuery, Cloud Storage, or Pub/Sub travels over an ALTS-protected path after GFE termination. Default protection is sufficient for many internal workloads, but additional controls are needed when internet ingress enters a VPC or when traffic leaves Google networks to an on-premises environment.

3.2Enforcing TLS for Application and API Traffic

Application-layer encryption adds TLS on top of default network protection for traffic to and from customer applications. Data flows from the end user to the load balancer to the backend VM or container, and each hop can require its own TLS decision. When an external Application Load Balancer or external proxy Network Load Balancer fronts an application, the connection between the end user and the load balancer is encrypted with TLS by default when the client supports TLS, and the GFE terminates that TLS using Google-managed or self-managed certificates.

To enforce HTTPS for applications and application programming interface (API) calls, use Cloud Load Balancing with HTTPS load balancers and managed certificates. Requests that clients send to a Google Cloud service over HTTPS, HTTP/2, or HTTP/3 are secured with TLS implemented in the GFE with BoringSSL, whose BoringCrypto core is validated to FIPS 140-3 Level 1 and negotiates forward-secure parameters with the client. For service-to-service authentication, use mutual TLS (mTLS) through Cloud Service Mesh for workloads on Google Kubernetes Engine (GKE) and Compute Engine, and TLS and mTLS configurations in Apigee for API traffic. Encryption from the load balancer to the backends must be enabled separately, and Cloud Run custom domains and Firebase Hosting can use free automated TLS certificates with options for customer certificates and HTTP Strict Transport Security (HSTS) headers.

3.3Keeping Managed Database Traffic Private with VPC Service Controls

VPC Service Controls creates a security boundary around Google-managed resources to keep private data flows inside a defined service perimeter and prevent data exfiltration. The perimeter contains selected projects, VPC networks, and services such as Cloud SQL, Spanner, BigQuery, and Cloud Storage, so data movement across the perimeter boundary is denied unless an ingress or egress rule explicitly allows it. Requests that reach Google Cloud services pass through identity authentication, Identity and Access Management (IAM) permission checks, and VPC Service Controls perimeter checks before the backend service is called:

1. Front-end receipt: The request arrives through the Google Front End service or Cloud Front End and is routed for authentication.
2. Identity verification: The central identity service authenticates the caller and issues short-lived context for the request.
3. Policy enforcement: The request is checked against IAM permissions, VPC Service Controls perimeters, quotas, and billing.
4. Backend invocation: Only after all checks pass does traffic flow from the enforcement point to the Google Cloud backend service.

To restrict inbound and outbound traffic to pipeline resources, configure firewall rules with policies that deny all traffic by default and explicitly allow only traffic that meets defined criteria. To permit access from on-premises networks, define allowlisted IP addresses in access level policy rules. Place sensitive data in its own perimeter, set IAM access controls on that data, and require multifactor authentication for users who need access.

3.4Adding Encryption for Hybrid Links to On-Premises Environments

Connections between Google Cloud and on-premises environments leave Google networks and require explicit transit encryption decisions. Cloud Interconnect provides a direct physical connection with high throughput for streaming ingestion, and this architecture recommends Dedicated Interconnect for large data volumes. Traffic over Cloud Interconnect is not encrypted in transit by default, and traffic to VMs over external IP addresses also leaves Google networks, so the customer is responsible for configuring encryption on those paths.

Use MACsec for Cloud Interconnect to encrypt traffic over interconnect connections and Cloud VPN with IPsec to encrypt traffic over virtual private network connections. Data flows from the on-premises gateway over the encrypted tunnel or MACsec-secured link into the VPC, where internal VM-to-VM and GFE-to-service protections then apply. Before moving sensitive data, encrypt it locally with client-side encryption so it stays protected at rest and in transit, using libraries such as Tink that are compatible with BigQuery Authenticated Encryption with Associated Data (AEAD) column decryption after import.

3.5Choosing a Transit Encryption Option

The three transit options differ by what path they protect and who configures them:

Layer 7 and hybrid encryption shift responsibility to application and network teams because they cannot be enforced centrally in the landing zone. Allow a limited exception during large migrations when legacy applications did not previously encrypt internal traffic on-premises and forcing modernization would unacceptably delay migration.

Key Facts

• Customer-managed encryption keys (CMEKs) in Cloud Key Management Service (Cloud KMS) act as key-encryption keys (KEKs) that wrap data-encryption keys (DEKs), and disabling or destroying the CMEK makes the protected data undecryptable.
• Cloud Storage applies a rotated CMEK only to newly written objects, while an existing BigQuery table keeps the key version assigned at creation time.
• Cloud KMS Autokey creates HSM-backed AES-256 GCM keys with yearly rotation in the same location as the protected resource.
• Traffic between virtual machines (VMs) inside a Virtual Private Cloud (VPC) and from the Google Front End (GFE) to Google Cloud services is encrypted by default with Application Layer Transport Security (ALTS), while Cloud Interconnect traffic requires MACsec or Cloud VPN with IPsec.
• VPC Service Controls keeps data flows inside a service perimeter around projects and services such as Cloud SQL, Spanner, BigQuery, and Cloud Storage, denying movement across the boundary unless an ingress or egress rule allows it.