Data Encryption and Key Management in GCP

Secure data processing pipelines with encryption protect data from ingestion through processing to storage and backup by combining default encryption, envelope encryption, and explicit transit controls. Data flows from on-premises or internet sources through processing engines such as Dataflow and Dataproc to storage services and backups, and every stage depends on encryption keys that are themselves encrypted and centrally managed. Cloud Key Management Service (Cloud KMS) (opens in a new tab) provides customer control over those keys when compliance requires more than Google-managed defaults.

1.1How Envelope Encryption Protects Pipeline Data

Envelope encryption in Google Cloud separates the key that encrypts data from the key that protects that key. A data encryption key (DEK) directly encrypts each logical chunk of customer content, while a key encryption key (KEK) wraps, or encrypts, the DEK itself for storage near the data. KEKs are stored centrally in Keystore, a repository built for storing keys, and one or more KEKs exist for each service rather than for each customer.

Data moves through envelope encryption in an ordered sequence:

1. Chunking: Incoming data is broken into logical chunks for storage, each up to several gigabytes in size and assigned a unique identifier.
2. Data encryption: Each chunk is encrypted at the storage level with its own individual DEK using AES-256 by default, so two chunks never share the same DEK.
3. Key wrapping: Each DEK is encrypted with a KEK and the wrapped DEK is stored near the data it encrypts to support low latency and high availability.
4. Access enforcement: Access control lists (ACLs) allow only Google services operating with authorized roles to decrypt a chunk at that point in time.
5. Decryption: To read customer data, a requester must obtain both all of the encrypted storage chunks and all of the corresponding unwrapped DEKs.

If a chunk is updated, it is encrypted with a new DEK rather than reusing the existing key, which limits the impact of a single DEK compromise to only that chunk. Each chunk is distributed and replicated in encrypted form for backup and disaster recovery, so backups never expose plaintext. Most backup files add another layer where each file is independently encrypted with its own DEK derived from a Keystore key and a randomly generated per-file seed.

1.2Layers of Default Encryption at Rest in Pipelines

Layers of default encryption at rest encrypt all customer content stored at rest without any customer action, using multiple redundant layers selected for application requirements. Customer content includes data stored in Cloud Storage (opens in a new tab) buckets, Persistent Disk volumes, and disk snapshots, while customer metadata such as project numbers, timestamps, and object sizes is protected separately for performance and operations. Encryption is inherent in storage systems and is applied before data is written to a database storage system or hardware disk.

The layers work together to remove lower hardware and software layers from the attack surface. At the infrastructure layer, storage-system encryption encrypts chunks with DEKs using AES-256, with the exception of a small number of Persistent Disks created before 2015 that use AES-128. At the storage-device layer, hard disk drives (HDD) and solid-state drives (SSD) apply a second AES-256 encryption with a separate device-level key that differs from the storage-level key. Google uses a FIPS 140-2 validated encryption module, certificate 4407, in production, and both AES-256 and AES-128 are recommended by NIST for long-term storage use. A logical chunk may contain data from multiple customers, so to achieve logical data separation through encryption, a pipeline must enable Cloud KMS with customer-managed keys rather than relying only on Google-owned and Google-managed keys.

1.3Choosing Pipeline Encryption at Rest for Compliance

Compliance requirements for pipeline encryption at rest determine whether default encryption is sufficient or whether customer-managed keys or tokenization are required. Default encryption shifts key lifecycle work to Google, while customer-managed options shift responsibility to application developers or a platform team to follow the security team's mandated key management. The security team can centrally enforce key control by blocking non-compliant resources with CMEK organization policies.

Tokenization cannot be centrally enforced in the landing zone and adds cost, latency, and complexity from extra encryption pipelines, so it should be avoided when Cloud KMS meets requirements. Customer-supplied encryption keys (CSEK) are not recommended because scenarios historically addressed by CSEK are now better addressed by Cloud EKM, which supports more services and higher availability. Cloud KMS Autokey automates provisioning and assignment of CMEKs, and CMEKs can be software-based or HSM-protected to meet rotation and regulatory needs.

1.4Key Lifecycle and Centralized Enforcement for Pipelines

Key lifecycle and centralized enforcement for pipelines covers generation, storage, rotation, status changes, and destruction of keys, key versions, and key rings. Control of keys means setting controls on the kind of keys and how keys are used, detecting variance, and planning corrective action, even when management is delegated to a third party through Cloud EKM. Centrally managed keys act as a chokepoint where access to data is enforced and can be audited, so protection can focus on keys instead of all data copies.

For processing pipelines, key control extends to where artifacts and intermediate data land. CMEK-integrated services encrypt data at rest on behalf of the pipeline when configured with customer-managed keys:

• Agent Platform
• Cloud Storage
• Cloud SQL
• Spanner
• BigQuery

Model artifacts, weights, files, and deployment configurations should be encrypted with Cloud KMS and stored in Cloud Storage with access controls or in Model Registry for fine-grained access control and encryption at rest. Server-side encryption helps protect CMEKs stored in Cloud Storage even if a self-managed key storage system is compromised.

1.5End-to-End Encryption in Transit for Pipelines and Hybrid Movement

End-to-end encryption in transit for pipelines and hybrid movement protects authenticity, integrity, and privacy while data moves outside physical boundaries not controlled by Google. Many internal pipeline patterns benefit from Google Cloud default encryption in transit. VM-to-VM traffic within a Virtual Private Cloud (VPC) network and between peered VPC networks is integrity-protected and encrypted inside Google's production network. Traffic from the internet or VPC network to Google services terminates at the Google Front End (GFE) (opens in a new tab), which provides DDoS countermeasures and routing, and Google's infrastructure then uses ALTS for authentication, integrity, and encryption from the GFE to a service and between services.

Pipelines that accept internet ingress or connect to on-premises environments need additional controls beyond defaults. Application-layer Layer 7 encryption protects traffic to customer applications through managed certificates on an external Application Load Balancer and through Cloud Service Mesh (opens in a new tab) and TLS and mTLS configurations for containerized and API traffic. All API calls to AI and ML services should use HTTPS, enforced with HTTPS load balancers. For hybrid paths, use MACsec for encryption over Cloud Interconnect connections and IPsec for traffic over Cloud VPN connections. End-to-end pipeline security also depends on boundaries around movement. A service perimeter created with VPC Service Controls (opens in a new tab) encloses AI, ML, and sensitive data resources inside a VPC boundary and controls data flow with ingress and egress rules to prevent exfiltration. Firewall rules that deny all traffic by default and explicitly allow only required traffic further restrict inbound and outbound access to processing resources.

Cloud Key Management Service (Cloud KMS) (opens in a new tab) is the service for creating and managing cryptographic keys used by Google Cloud services and by customer applications. Cloud KMS generates software or hardware key material, imports existing customer key material, or links external keys held in a compatible external key management system. Keys managed in Cloud KMS act as key-encrypting keys that wrap data-encrypting keys, sign data, or establish shared secrets, while access and compliance are controlled through Identity and Access Management (IAM) and organization policy.

2.1Key Hierarchy in Cloud KMS

The Cloud KMS key hierarchy organizes cryptographic material as key rings that contain keys, where each key has one or more key versions that hold the actual cryptographic material. A key ring is a regional or global container that groups related keys for a workload, and the key defines purpose and algorithm while key versions allow rotation without changing the key identity. Applications and Google Cloud services reference the key, and Cloud KMS routes encryption, decryption, signing, or verification operations to the designated primary version. Cloud KMS Autokey automates this hierarchy by generating key rings, keys, and service accounts on demand during resource creation, so teams do not provision them ahead of time.

Cloud KMS keys support multiple cryptographic operations beyond storage encryption. Data flows from the caller to Cloud KMS for centralized operations, and Cloud KMS returns ciphertext, plaintext, signatures, or shared secrets without exposing the key material itself:

• Encryption and decryption for client-side encryption through the Cloud KMS API and client libraries.
• Digital signatures for proving authenticity and integrity of data or messages.
• Message authentication code (MAC) signatures for verifying data with a shared secret key.
• Key encapsulation mechanisms for establishing shared secrets between parties.

2.2Protection Levels and Key Origins in Cloud KMS

Protection levels and key origins in Cloud KMS define where key material is generated and stored, which determines compliance, control, and operational overhead. Software keys are generated and stored in Google Cloud software, hardware keys are generated and protected in a Cloud HSM (opens in a new tab) (hardware security module), and external keys remain outside Google Cloud in a customer or partner external key management (EKM) system. Key import supports bring-your-own-key scenarios by importing customer-generated material into Cloud KMS or into Single-tenant Cloud HSM for use as a Cloud KMS key version.

Cloud HSM has two tenancy models with different isolation boundaries. Multi-tenant Cloud HSM keys share underlying hardware across tenants and are used with Cloud HSM for Google Workspace to enable client-side encryption (CSE) in Google Workspace. Single-tenant Cloud HSM instances provide dedicated hardware per customer, and the customer creates or imports Single-tenant Cloud HSM keys inside that instance before Cloud KMS uses those keys for operations.

2.3Envelope Encryption with Customer-Managed Encryption Keys

Envelope encryption with customer-managed encryption keys uses a customer-controlled Cloud KMS key as the key-encrypting key for service data. Customer-managed encryption keys (CMEK) are Cloud KMS keys that a customer controls and assigns to Google Cloud resources to protect data at rest. In the envelope encryption model, the Google Cloud service generates a data encryption key (DEK) to encrypt customer data, then sends the DEK to Cloud KMS where the CMEK as key-encrypting key (KEK) wraps, or encrypts, the DEK. The service stores the wrapped DEK alongside the encrypted data, and must call back to Cloud KMS to unwrap the DEK before data can be decrypted, so revoking or disabling the CMEK blocks access to the data.

CMEK integrations use this same wrap and unwrap flow across data services including BigQuery, Cloud Storage, Cloud SQL, Spanner, Pub/Sub, Compute Engine persistent disks, Dataflow, Dataproc, and Vertex AI. The service identity that performs wrapping depends on Cloud KMS for every encrypt and decrypt operation, which creates a dependency between the data resource and the key ring, key, and key version. Choose CMEK over Google-owned and Google-managed default encryption when there is a requirement to manage the key lifecycle, to prove key management to external auditors, to use FIPS 140-2 Level 3 hardware, or to keep key material outside Google Cloud.

2.4Rotation, Destruction, and Lifecycle Governance for Keys

Rotation, destruction, and lifecycle governance for keys covers generation, use, rotation, retirement, and destruction of keys and key versions to meet cryptoperiod and compromise requirements. Rotation creates a new primary key version for future encryption while prior versions remain available to decrypt previously protected data, so rotation limits the exposure of any single version without causing data loss. Retirement, replacement, or destruction is required when a key reaches the end of its defined cryptoperiod, when its integrity is weakened, when compromise is suspected, or when personnel with knowledge of cleartext material change roles, and retired keys must not be used for new encryption operations.

A complete lifecycle program tracks all cryptographic materials and status changes for audit and regulatory review. Controls that enforce this program verify the Cloud KMS protection level of each key and restrict allowed key version algorithms, such as approved RSA signing and decryption algorithms, to ensure modules, algorithms, and key lengths match risk strategy. Security teams enforce consistent use by blocking creation of non-compliant resources with CMEK organization policies, so developers can only deploy resources that reference approved Cloud KMS keys.

2.5Access Control for Cloud KMS Keys with IAM

Access control for Cloud KMS keys with IAM uses IAM to grant only the minimum permissions needed to use, manage, or administer keys. IAM roles assigned to users, groups, or service accounts separate key administrators who create key rings and set rotation from key users such as service agents that only call encrypt and decrypt. The service that hosts CMEK-protected data depends on an IAM grant on the key, so data flows only after the service identity receives permission to use the specific key for wrapping and unwrapping.

Least-privilege enforcement extends beyond direct grants to conditions and boundaries around key use. IAM policies with conditions restrict key use by attributes such as time, resource type, or request context, while principal access boundary policies and just-in-time elevation limit which principals can obtain powerful key administration rights. Combined with CMEK organization policies that deny resources without customer-managed keys, IAM ensures that key custodianship stays with the fewest necessary custodians and that every use of a key is attributable through audit logs.

Google Cloud (opens in a new tab) encryption protects customer content stored at rest by default with no customer action, while data in transit is encrypted and authenticated when it moves outside physical boundaries controlled by Google or on behalf of Google. Google Cloud divides responsibility by state: storage systems and hardware enforce at-rest encryption automatically, network layers enforce in-transit encryption by default, and Cloud Key Management Service (Cloud KMS) (opens in a new tab) provides customer control over keys when compliance requires it. Choosing an implementation means deciding whether default encryption is sufficient, whether customer-managed keys are required, and whether application-layer or link-layer controls are needed for transit.

3.1How Default Encryption at Rest Works in Google Cloud

How default encryption at rest works in Google Cloud is through multiple redundant layers that encrypt all customer content before it is written to a database storage system or hardware disk. At the infrastructure layer, data is broken into logical chunks up to several gigabytes in size, and each chunk is encrypted with its own data encryption key (DEK) using AES-256, with the exception of a small number of Persistent Disks created before 2015 that use AES-128. If a chunk is updated, it is encrypted with a new key rather than reusing the existing key, so compromise of one DEK is limited to only that chunk.

The infrastructure layer uses envelope encryption to protect DEKs at scale with low latency and high availability. DEKs are stored near the data they encrypt but are themselves encrypted, or wrapped, by a key encryption key (KEK), and those KEKs are stored centrally in Keystore, a repository built for storing keys. Each chunk has a unique identifier and access control lists (ACLs) ensure each chunk can be decrypted only by Google services operating with authorized roles granted only at that point in time, and chunks are distributed and replicated in encrypted form for backup and disaster recovery. Protection continues below and beyond the storage system. A separate device-level key encrypts hard disk drives (HDD) and solid-state drives (SSD) with AES-256, different from the storage-level key, and backups remain encrypted throughout the backup process with most backup files independently encrypted with their own DEK derived from a Keystore key and a per-file seed. Google uses a FIPS 140-2 validated encryption module, certificate 4407, in production, and both AES-256 and AES-128 are recommended by the National Institute of Standards and Technology (NIST) for long-term storage use. A logical chunk can contain data from multiple customers, so to achieve logical separation through encryption you must enable Cloud KMS.

3.2Key Management Options for Data at Rest in Google Cloud

Key management options for data at rest in Google Cloud range from fully Google-managed to fully external, with increasing control and increasing operational overhead. Cloud KMS lets you create software or hardware keys, import existing keys, or link external keys, and then use customer-managed encryption keys (CMEKs) in CMEK-integrated services where your key wraps the service DEK. Cloud HSM provides hardware-backed keys, Cloud External Key Manager (Cloud EKM) keeps key material outside Google Cloud in a compatible external key management system, and customer-supplied encryption keys (CSEKs) are keys you supply per object that are not recommended because Cloud EKM supports more services with higher availability.

Use default encryption when you have no particular requirements around how data is encrypted or how keys are managed and you prefer to avoid key-management cost and overhead. Use CMEK when you must manage the lifecycle of your own keys, generate material in a FIPS 140-2 Level 3 HSM, or store material outside Google Cloud with Cloud EKM, and enforce it by blocking non-compliant resource creation with CMEK organization policies (opens in a new tab) such as gcp.restrictNonCmekServices. Avoid tokenization at the application layer, for example ingesting with Sensitive Data Protection to tokenize personally identifiable information (PII) before persisting to BigQuery, when Cloud KMS is sufficient, because added encryption pipelines add cost, latency, and complexity and cannot be centrally enforced in the landing zone.

3.3Implementing CMEK for Storage, BigQuery, SQL, and Spanner

Implementing CMEK for storage, BigQuery, SQL, and Spanner means associating your Cloud KMS key with CMEK-integrated services that then encrypt data at rest for you. CMEK-integrated services include Cloud Storage (opens in a new tab), BigQuery, Cloud SQL, and Spanner. In each service the flow is the same: data flows from the client to the service storage layer, the service generates a DEK for the data chunk, and Cloud KMS wraps that DEK with your KEK using envelope encryption. The service depends on Cloud KMS and Identity and Access Management (IAM) for every decrypt, so revoking or destroying the KEK makes the wrapped data unreadable and creates a central chokepoint where access can be enforced and audited.

Implementation details differ by service but share location and lifecycle rules. You must create the key ring and key in the same location as the bucket, dataset, instance, or database, because the CMEK location determines where the key is stored and made available for access. Cloud KMS Autokey automates provisioning and assignment so key rings, keys, and service accounts are generated on demand during resource creation rather than ahead of time. For BigQuery, you configure a dataset default key for the warehouse, and you can add column-level controls with policy tags, dynamic data masking, and AEAD column encryption where the wrapped DEK and data are stored in separate tables and unwrapped with the KEK at query time.

3.4Encryption in Transit by Default and When to Add More

Encryption in transit by default in Google Cloud authenticates, provides integrity, and encrypts connections that leave Google-controlled physical boundaries. Traffic from the internet to Google services and from a Virtual Private Cloud (VPC) network to Google services terminates at the Google Front End (GFE), which provides distributed denial-of-service (DDoS) countermeasures and routes and load-balances traffic to services. Connections from the GFE to a Google Cloud service and from one Google Cloud service to another use Application Layer Transport Security (ALTS) for authentication, integrity, and encryption, and virtual machine (VM)-to-VM traffic within a VPC network and between peered VPC networks inside the production network is integrity-protected and encrypted.

Default transit protection is sufficient for many internal workloads, but add controls for internet ingress and hybrid connections. For application traffic, configure Layer 7 encryption that developers own per workload, such as HTTPS for all API calls to AI and machine learning (ML) services, external Application Load Balancers with Google-managed SSL certificates, and mutual Transport Layer Security (TLS) with Cloud Service Mesh (opens in a new tab) and Apigee ingress TLS for service-to-service calls. For connections to on-premises environments, traffic over Cloud Interconnect is not encrypted in transit by default, so use MACsec for encryption over Interconnect and IPsec for encryption over Cloud VPN, with Dedicated Interconnect recommended when streaming large volumes for its direct connection and high throughput.

Key Facts

• Envelope encryption encrypts each storage chunk with its own data encryption key (DEK) using AES-256, then wraps each DEK with a key encryption key (KEK) stored centrally in Keystore.
• Customer-managed encryption keys (CMEKs) in Cloud KMS let customers control key lifecycle, and revoking or disabling the CMEK makes data encrypted with the wrapped DEKs unreadable.
• Cloud HSM generates and protects keys in FIPS 140-2 Level 3 certified hardware, while Cloud External Key Manager (Cloud EKM) keeps key material outside Google Cloud.
• Traffic over Cloud Interconnect is not encrypted by default and requires MACsec, while traffic over Cloud VPN uses IPsec.
• Default encryption at rest uses AES-256 for storage-system and device-level layers, except for a small number of Persistent Disks created before 2015 that use AES-128.