professional-cloud-data-engineer
Encryption for data in transit protects the confidentiality and integrity of data as it moves between systems. In Google Cloud, this involves establishing secure communication channels to prevent unauthorized access or tampering during transmission. While Google's infrastructure automatically encrypts traffic within its physical boundaries, customers are responsible for ensuring encryption is applied when data moves externally or to specific managed services.
The primary method for securing data in transit is enforcing Transport Layer Security (TLS). For managed database services like Cloud SQL and Spanner, you must explicitly configure client connections to use TLS, ensuring all client-to-database traffic is encrypted. For proxy load balancers such as the Global External Application Load Balancer, you should select HTTPS or HTTP/2 as the backend service protocol. This provides end-to-end encryption from the load balancer to your backend instances, going beyond the automatic network-level encryption provided within Google's VPC networks.
VPC Service Controls creates security perimeters around Google-managed services like BigQuery or Cloud Storage to prevent data exfiltration. These perimeters define private data flows and logically isolate the services within your cloud network. To secure connections into these perimeters from on-premises or other clouds, you use Cloud Interconnect or Cloud VPN. These services establish encrypted tunnels over the public internet or dedicated private connections, protecting all ingress and egress traffic to your cloud environment.
A complete strategy for data-in-transit encryption on Google Cloud involves a layered approach:
Google Cloud provides multiple encryption options for data at rest, each offering different levels of customer control. By default, Google encrypts all customer content using Google-owned and Google-managed encryption keys, which requires no action from customers. However, organizations with compliance requirements may need more control over their encryption keys.
Customer-managed encryption keys (CMEK) allow you to manage encryption keys using Cloud Key Management Service (Cloud KMS) . With CMEK, you create and control the keys that protect your data, while Google handles the encryption and decryption operations. This option is ideal for organizations that need to demonstrate key management control to auditors or comply with regulatory requirements. CMEK can be applied to numerous Google Cloud services including BigQuery datasets, Cloud Storage buckets, Compute Engine persistent disks, Dataproc clusters, and Pub/Sub topics. You can choose between software keys, hardware security module (HSM) keys for FIPS 140-2 Level 3 compliance, or external keys via Cloud External Key Manager (Cloud EKM) for data sovereignty requirements.
Customer-supplied encryption keys (CSEK) provide the highest level of customer control by allowing you to generate and manage encryption keys entirely outside of Google Cloud. These keys are supplied by the customer when creating or accessing certain resources. However, CSEK is generally not recommended for new implementations—scenarios historically addressed by CSEK are now better served by Cloud External Key Manager (Cloud EKM) because it offers support for more services and higher availability. Cloud EKM lets you store cryptographic key material outside of Google Cloud while still integrating with Google Cloud services, giving you control over where keys are stored geographically.
When implementing CMEK, you must consider key location and rotation. The CMEK location determines where the key is stored and must match the location of your resources. Google recommends enabling automatic key rotation, which by default occurs every 30 days. For data sovereignty requirements, Cloud EKM allows you to store keys outside of Google Cloud, meeting geographic control requirements.
Some services are CMEK-compliant, meaning they either do not store data or only store data briefly during processing, using ephemeral keys that exist only in memory and are never written to disk. Understanding which services support CMEK versus which are CMEK-compliant helps you design appropriate data protection strategies for your pipelines.
Cloud Key Management Service (Cloud KMS) is the central service for implementing customer-managed encryption keys (CMEK) in Google Cloud. It provides a provable and monitorable root of trust, allowing organizations to manage key rings, keys, and key versions. When planning data pipelines, choosing between CMEK, Cloud KMS Autokey, and customer-supplied encryption keys (CSEKs) is vital to align with organizational governance. Each choice impacts costs, administrative complexity, and operational transparency.
Data engineers can configure CMEKs to protect a variety of Google Cloud services used in data engineering pipelines. The following services support CMEK for encryption at rest:
For database-specific pipelines like Cloud SQL, a dedicated service agent must be granted the Cloud KMS Encrypter/Decrypter role to access the required key.
Enforcing a strict key rotation policy is a fundamental administrative practice. Data engineers should establish automatic rotation schedules in Cloud KMS to regularly update keys and limit the impact of potential key compromises. However, original primary key versions must never be deleted, because destroying them prevents the restoration of older database backups. Additionally, regionality is a critical design constraint: regional resources like Cloud SQL instances and persistent disks can only be encrypted by keys located in the matching region.
Enterprise data pipelines often require stringent auditing and cloud provider access controls. Using Cloud External Key Manager (Cloud EKM) , organizations can store encryption keys externally while maintaining data separation from Google Cloud infrastructure. To audit and control access to these keys, administrators can implement advanced security controls:
This layered defense ensures that the customer remains the ultimate arbiter of access to their sensitive pipeline data.
Prepare and test your skills
Prepare and test your skills
CMEK allows you to manage encryption keys using Cloud Key Management Service (Cloud KMS), while CSEK lets you generate and manage keys entirely outside of Google Cloud. CSEK is not recommended for new implementations because Cloud External Key Manager (Cloud EKM) supports more services and offers higher availability.
Google recommends enabling automatic key rotation, which by default occurs every 30 days. However, original primary key versions must never be deleted because destroying them prevents the restoration of older database backups.
For Cloud SQL, a dedicated service agent must be granted the Cloud KMS Encrypter/Decrypter role to access the required key. Additionally, the CMEK location must match the location of the Cloud SQL instance since regional resources can only be encrypted by keys located in the matching region.
Cloud EKM provides three advanced controls: Access Transparency generates near-real-time logs of administrative actions taken by Google personnel; Access Approval requires explicit manual consent before Google operators can access data or configurations; and Key Access Justifications provides a programmatic reason for each key request, allowing automated policies to block or permit decryption.
A data engineering team is designing an automated ingestion pipeline that processes sensitive transaction files exceeding 50 MB before storing them in Google Cloud. The organization's security policy requires column- and application-layer encryption managed centrally through Cloud Key Management Service (Cloud KMS).
During architectural review, the team observes that the Cloud KMS Encrypt API rejects direct encryption requests exceeding 64 KiB.
Which cryptographic key management strategy should the team implement to encrypt these large payloads securely and efficiently?
StudyCerts
StudyCerts