Designing for security and compliance

Designing for Security and Compliance

This section covers the core security, privacy, and compliance concepts that a Professional Cloud Data Engineer must understand to design secure data systems on Google Cloud. It addresses identity and access management, data encryption, privacy protection for sensitive information, and regional data sovereignty requirements.

Identity and Access Management (e.g., Cloud IAM and organization policies)

Least-Privilege IAM Frameworks for Data Workloads

Least-privilege access is the principle of granting users and service accounts only the minimum permissions required for their tasks. In Google Cloud, this is managed through Cloud Identity and Access Management (IAM), which controls access to resources across all services. IAM focuses on resource-level access control, assigning permissions to users, groups, and service accounts for specific Google Cloud resources, thereby reducing the overall attack surface.

Google Cloud provides three main types of roles to implement least privilege. Basic roles (Owner, Editor, Viewer) are broad and generally not recommended for strong security. Predefined roles offer granular permissions tailored to specific services like BigQuery and Cloud Storage and are designed for common use cases. For unique requirements, custom roles can be created by bundling specific permissions together. For BigQuery, additional fine-grained controls include table-level, column-level (using policy tags), and row-level security.

Service account impersonation is a secure practice that eliminates the need for long-lived credential sharing in automated pipelines. Instead of using service account keys, authorized users can temporarily impersonate a service account. This requires the user to have either the Service Account User role (to act as the service account) or the Service Account Token Creator role (to generate short-lived access tokens). Impersonation creates temporary tokens, improving security. Where possible, Workload Identity Federation is a more secure alternative, allowing external workloads to impersonate service accounts without any keys.

Implementing Fine-Grained Access and Context-Aware Security

VPC Service Controls mitigate data exfiltration risks by creating security perimeters, called service perimeters, around Google-managed services like BigQuery and Cloud Storage. These perimeters control data movement across the boundary. To allow authorized communication between separate projects within different perimeters, administrators can configure perimeter bridges.

BigQuery provides native tools for granular data security. Column-level access control uses policy tags to restrict access to specific columns, requiring users to have the Fine-Grained Reader role. Row-level security filters dataset rows based on the user's identity or other conditions. Dynamic data masking selectively obscures column values when a query runs, based on the user's roles. For cell-level encryption, AEAD encryption functions require the user to have permissions for both the table and the cryptographic key to decrypt data.

Enforcing Organization Policies and Resource Hierarchy Constraints

Organization Policies are centralized, declarative rules that enforce security, compliance, and governance guardrails across your Google Cloud resource hierarchy (Organization, Folders, Projects). They act as preventive controls to establish a consistent security baseline.

A core feature is policy inheritance. A policy set at a higher level (e.g., the Organization node) automatically applies to all child Folders and Projects. However, you can override an inherited policy by setting a different, more specific policy directly on a child resource. This allows for broad compliance enforcement while permitting justified exceptions, such as allowing a specific project to use additional geographic regions.

Several critical organization policy constraints are used to establish foundational security:

• gcp.resourceLocations: Restricts the geographic regions where resources (like Compute Engine instances or Cloud Storage buckets) can be created, helping meet data residency requirements.
• iam.disableServiceAccountKeyCreation: A boolean constraint that prevents the creation of external, long-lived service account keys, forcing the use of more secure methods like impersonation or Workload Identity Federation.
• iam.automaticIamGrantsForDefaultServiceAccounts: Disables the automatic granting of broad roles (like Editor) to default service accounts, enforcing the principle of least privilege from creation.

For granular, attribute-based control, you can use tags. Tags are metadata labels attached to resources. You can then create organization policies or IAM policies that apply conditionally based on these tags. For example, you could allow Public IP addresses only for resources tagged as "public-facing," or enforce stricter constraints on resources that lack a specific "exempt" tag.

Data security (encryption and key management)

Design and Manage Cryptographic Key Lifecycles Using Cloud KMS and HSM

Cloud Key Management Service (Cloud KMS) and Cloud HSM are Google Cloud's services for creating, storing, and managing cryptographic encryption keys throughout their lifecycle. Cloud HSM provides hardware security modules that meet FIPS 140-2 Level 3 certification for organizations with strict compliance needs. Both services implement envelope encryption, where a data encryption key (DEK) encrypts the data itself, and a separate key encryption key (KEK) encrypts (wraps) the DEK. This hierarchy allows Google to manage large numbers of keys securely.

The key management hierarchy in Cloud KMS has three levels. A key ring is a logical container that groups related crypto keys together, often organized by application or environment. Each crypto key can have multiple key versions, where each version is a distinct cryptographic key that can be in an active, pending, or disabled state. When a new version is created, it becomes the primary version for encryption, while older versions remain available only for decryption of data they previously encrypted. This versioning approach supports key rotation without data loss, because both old and new versions remain functional for their respective purposes.

Key rotation limits the amount of data protected by a single key, reducing the potential impact of key compromise. Cloud KMS supports both automatic and manual rotation schedules; Google recommends a rotation period of typically 90 days for most use cases. When rotation occurs, a new key version is automatically created, and the previous version remains available for decryption. The full key lifecycle extends from generation through rotation to destruction. Organizations must define rotation periods, establish destruction procedures, and maintain audit trails of all key operations to prevent data loss while maintaining security.

Access to cryptographic keys is controlled through Identity and Access Management (IAM) policies. Google recommends limiting the number of principals who can use cryptographic keys to three or fewer. The principle of separation of duties means that users who manage keys should be different from those who use them for encryption and decryption. Cloud KMS provides predefined roles such as Cloud KMS Admin, CryptoKey Encrypter/Decrypter, and CryptoKey Viewer to implement granular access control. Organizations should create separate projects for key management and data storage to strengthen security boundaries and ensure that key administrators cannot directly access the data those keys protect.

Customer-managed encryption keys (CMEK) give organizations full control over their encryption keys while using Google Cloud's infrastructure for key operations. CMEK integrates with services such as BigQuery, Cloud Storage, Compute Engine, Spanner, and Vertex AI. For compliance, Cloud KMS supports FIPS 140-2 validated hardware through Cloud HSM. The Cloud External Key Manager (Cloud EKM) feature allows organizations to keep key material in customer-controlled environments, addressing data sovereignty requirements.

Cloud KMS Autokey automates the provisioning of customer-managed encryption keys when resources are deployed. Key rings and keys are generated on-demand, and service accounts are automatically granted the necessary IAM roles for encryption and decryption. Autokey follows Google's recommended practices, including using the Cloud HSM protection level, proper key granularity, and appropriate rotation schedules.

Evaluate and Implement Encryption Methods for Data at Rest and in Transit

Encryption in Google Cloud is applied by default to protect data both at rest and in transit. For data at rest, Google uses multiple layers of encryption, including storage system-level and storage device-level encryption. Each logical chunk of data is encrypted with a unique data encryption key (DEK), which is then wrapped by a key encryption key (KEK) stored centrally – this is envelope encryption. Data in transit is automatically encrypted when it moves outside Google's physically secured network boundaries, and internal traffic between Google Cloud services also uses encryption with integrity protection.

Customers have several options for managing encryption keys, balancing control with operational overhead:

When implementing encryption for specific Google Cloud services, each service has its own capabilities. Cloud Storage, BigQuery, Cloud SQL, and Spanner all support CMEK for data at rest. Data can be further protected using client-side encryption libraries such as Tink before ingestion, or via column-level encryption within BigQuery. For data in transit, services can be configured to enforce TLS, and network traffic can be secured using Cloud VPN, Cloud Interconnect, or VPC Service Controls to define secure perimeters.

Secure Data Processing Pipelines with Encryption and Envelope Encryption

Securing data processing pipelines in Google Cloud requires a defense-in-depth approach using envelope encryption. Data is encrypted with a Data Encryption Key (DEK), which is then wrapped and protected by a Key Encryption Key (KEK) managed centrally in Cloud Key Management Service (Cloud KMS) or Cloud HSM. This layered strategy ensures that if an attacker compromises a physical storage system, they cannot read the underlying information without authorized access to both the wrapped key and the central key manager.

For analytical workloads, security is implemented directly within data processing engines. Cloud Dataflow and Cloud Dataproc both support Customer-Managed Encryption Keys (CMEK), allowing organizations to maintain full lifecycle control over the keys used to encrypt pipeline states, temporary files, and virtual machine disks. During the ingestion phase, Dataflow pipelines can integrate with Sensitive Data Protection to tokenize, de-identify, or encrypt individual database columns before storing them in BigQuery.

When importing sensitive records from external or on-premises environments, client-side encryption ensures end-to-end protection. Organizations can use the Tink encryption library to secure confidential column values locally before sending them over the network. To secure the communication path itself, Cloud VPN or Cloud Interconnect establishes an encrypted, private tunnel to Google Cloud. This combined approach ensures that data remains unreadable to unauthorized entities from the moment it leaves the local network until it is processed in the cloud.

Administering key lifecycles is critical for maintaining a strong security posture. The standard lifecycle management workflow involves the following ordered steps:

1. Create Keys: Generate cryptographically strong keys using certified hardware security modules.
2. Assign Authority: Restrict key access to specific service accounts based on the principle of least privilege.
3. Rotate Keys: Schedule automatic key rotation every thirty days to limit potential data exposure.

Privacy (e.g., strategies to handle personally identifiable information)

Application of Transformational De-identification Techniques

Transformational de-identification modifies sensitive records such as personally identifiable information (PII) so organizations can analyze data without exposing original identities. Google Cloud provides these capabilities through Sensitive Data Protection, which replaces sensitive data points with protected tokens or generalized values. Two-way tokenization methods, such as format-preserving encryption (FPE) and deterministic encryption (DE), replace raw values with encrypted values using a key. These two-way techniques preserve referential integrity, which allows analytical systems to join related records across tables without exposing the underlying plaintext. In contrast, one-way tokenization uses cryptographic hashing to generate irreversible tokens, making it ideal for joining records when original values never need to be decrypted.

Non-cryptographic transformations obscure or generalize data values to reduce re-identification risks without using cryptographic keys. Masking replaces sensitive characters with a fixed symbol, such as an asterisk, hiding identifiers while partially displaying non-sensitive fragments. Bucketing groups exact numerical or demographic values into generalized ranges, such as converting specific ages into ten-year brackets. Date shifting moves recorded timestamps by a consistent, pseudorandom offset per individual, preserving the time intervals between events while masking actual calendar dates. Choosing between these techniques requires balancing data utility against privacy preservation, because aggressive masking or bucketing reduces the statistical detail available to analysts.

Scalable de-identification architectures use automated data processing pipelines to clean data before it reaches analytical storage. In this workflow, raw data streams into Cloud Dataflow, which processes batch or streaming records by executing transformation rules defined in Sensitive Data Protection templates. These templates decouple transformation logic from application code, allowing security teams to manage policies centrally and reuse them across multiple pipelines. Cryptographic techniques depend on a token encryption key managed through Cloud Key Management Service (Cloud KMS) as a wrapped key. Administrators regularly rotate these encryption keys and assign separate keys to distinct data fields to prevent a single compromised key from exposing all datasets.

Granular Access Control and Governance for PII Protection

Granular access controls in analytical warehouses restrict data visibility based on user identity, column sensitivity, and row attributes. BigQuery secures sensitive attributes through column-level security using policy tags managed in Dataplex Universal Catalog (formerly Data Catalog). When a policy tag is applied to a column, only users with specific Identity and Access Management (IAM) roles can view the unmasked data. BigQuery also supports row-level access policies, which filter query results based on user identity, and authorized views, which grant query access to aggregated results without exposing the underlying base tables. Dynamic data masking works alongside column-level security to let authorized users query non-sensitive columns while automatically showing masked or hashed values for protected columns.

Enterprise data governance combines automated metadata tracking with virtual perimeters to prevent unauthorized data access and exfiltration. Administrators configure VPC Service Controls (VPC SC) to build a security perimeter around resources like BigQuery and Cloud Storage, blocking unauthorized data transfers to external environments. Security policies can also enforce context-aware access, limiting service account and user operations to verified corporate IP addresses and managed devices. To maintain continuous governance across large data estates, administrators implement the following automated practices:

• Automated scanning: Discovers and flags sensitive financial, health, or identity records across all storage locations.
• Policy tagging: Attaches taxonomy tags and security access rules to dataset metadata automatically upon discovery.
• Retention control: Executes lifecycle rules that automatically archive or purge sensitive files based on compliance schedules.

Regulatory compliance requires organizations to track how sensitive data moves through the cloud and monitor every user interaction. Google Cloud captures administrative and data-access operations using Cloud Audit Logs and Access Transparency logs, providing an immutable record of user and service actions. Dataplex tracks data lineage automatically, recording where datasets originate, what transformations run against them, and which downstream tables or dashboards consume them. This lineage visibility allows compliance teams to trace the flow of sensitive data, verify that protection policies were applied during processing, and detect anomalous data access patterns.

Automated Detection and Classification of Sensitive Data

Automated detection identifies and categorizes sensitive data across structured databases, data warehouses, and object storage. Sensitive Data Protection uses built-in infoTypes, which are detectors configured to identify specific data categories such as credit card numbers, Social Security numbers, passport IDs, and medical records. These predefined detectors analyze data patterns using regular expressions, character context, and mathematical checksum algorithms to reduce false positives. When organizations handle proprietary internal formats, such as custom customer identification numbers or employee IDs, administrators create custom infoTypes using regular expressions, word dictionaries, or stored lookup tables.

Google Cloud provides two complementary scanning mechanisms to discover and monitor sensitive data at scale:

1. Sensitive data profiling: Scans entire organizations, folders, and projects automatically to generate high-level risk profiles and identify tables containing sensitive data.
2. On-demand inspection: Executes targeted, in-depth scans against specific BigQuery tables, Cloud Storage buckets, or relational databases to identify precise cell and column locations of sensitive values.
3. Automated transformation: Triggers de-identification actions, such as redaction, masking, or tokenization, immediately after an inspection job detects sensitive records.

Inspection templates standardize detection rules across continuous data ingestion pipelines and batch scanning jobs. These templates define the exact infoTypes to detect, the likelihood thresholds required to trigger a match, and the maximum number of findings to record per scan. Security teams manage access to these templates using IAM roles, ensuring that developers and data engineers can apply standardized classification rules without altering security configurations. When integrated into ingestion pipelines with Cloud Dataflow or Cloud Storage, inspection templates ensure that all incoming data is classified consistently before it is committed to analytical storage.

Regional considerations (data sovereignty) for data access and storage

Enforcing Data Sovereignty via Google Cloud Organization Policies

Data sovereignty requires that digital information adheres to the legal frameworks and privacy regulations of the geographic location where it is collected, stored, and processed. In Google Cloud, administrators enforce data sovereignty across resource hierarchies by applying the resource location restriction constraint (gcp.resourceLocations) within the Organization Policy Service. This constraint restricts the provisioning of data storage and compute resources to an approved list of regions, multi-regions, or zones. When applied at the organization, folder, or project level via Resource Manager, the policy blocks any attempt to create non-compliant resources while permitting existing resources to continue running.

Organizations can define allowed locations using granular individual zones, specific regions such as europe-west1, or multi-region identifiers like eu and us. To simplify administration across multiple compliant territories, administrators can also apply predefined location value groups, such as in:eu-locations or in:us-locations. Beyond custom policies, Assured Workloads automatically enforces pre-configured regional boundaries and resource constraints for specific compliance regimes, such as the EU Data Boundary, US Data Boundary, Japan Data Boundary, and Australia Data Boundary. These controls help satisfy standard security frameworks, including NIST-800-53 controls for access enforcement (AC-3), remote access (AC-17), and information sharing (AC-20).

Organizations must also account for the functional boundaries of resource location constraints. Global resources, such as Identity and Access Management (IAM) policies and global load balancers, operate outside regional boundaries and cannot be restricted by gcp.resourceLocations. To maintain strict sovereignty, security teams combine location constraints with the gcp.restrictServiceUsage constraint to block unauthorized cloud services within the environment.

Configuring Regional, Dual-Regional, and Multi-Regional Storage Topologies

Data residency requires storing and managing data within designated physical borders to comply with legal mandates like the General Data Protection Regulation (GDPR). When designing storage architectures in Google Cloud using services like Cloud Storage, BigQuery, and Cloud Spanner, engineers select specific regional topologies that balance legal restrictions against availability, cost, and disaster recovery needs. Choosing a topology dictates the physical boundaries of data at rest, replication paths, and recovery capabilities.

Storage topologies vary by the geographic distribution of their underlying infrastructure:

To preserve data residency during disaster recovery operations, secondary backup targets must remain inside compliant jurisdictions. For instance, Cloud SQL cross-region read replicas must be deployed strictly to secondary regions that satisfy regulatory limits, and BigQuery datasets should export data exclusively to co-located or legally compliant Cloud Storage buckets. Additionally, BigQuery provides dedicated regional API endpoints, such as bigquery.us-central1.rep.googleapis.com, which direct all request metadata and in-transit data exclusively through servers in that specific region.

Governing Regional Processing, Network Egress, and Cryptographic Key Residency

Regional data processing mandates that compute workloads, such as Dataproc clusters and Dataflow pipelines, operate in the exact same physical region where their input and output data resides. If a data pipeline reads from a regional Cloud Storage bucket in europe-west1, the Dataproc or Dataflow compute instances must execute within europe-west1 to prevent unauthorized cross-border network transfers. This geographical alignment ensures that intermediate shuffle data, temporary execution files, and analytical outputs never leave the defined sovereignty boundary.

Cryptographic security must also respect data sovereignty through localized customer-managed encryption keys (CMEK) managed by Cloud Key Management Service (Cloud KMS). While Google Cloud automatically manages low-level data encryption keys (DEKs), organizations maintain control over the key encryption keys (KEKs) by hosting the CMEK in the identical region as the storage or compute resource being encrypted. To enable automated encryption and decryption across processing workflows, the Cloud KMS CryptoKey Encrypter/Decrypter role must be granted to the following service agents:

• Compute Engine service agent
• Cloud Storage service agent
• Dataproc service agent

To prevent data exfiltration across network boundaries, organizations deploy VPC Service Controls (VPC SC) to create secure service perimeters around cloud resources. VPC Service Controls restrict communication to authorized networks and prevent data from moving outside the perimeter without explicit ingress and egress policy rules. To ensure network traffic stays on private Google networks and avoids public exposure, internal Domain Name System (DNS) records route API requests to the restricted.googleapis.com domain, which restricts communication only to Google APIs supported by VPC Service Controls.

Legal and regulatory compliance

This area covers understanding and applying relevant legal and regulatory frameworks to data engineering solutions on Google Cloud. Data engineers must be familiar with regulations such as the General Data Protection Regulation (GDPR), health data regulations like HIPAA, financial data requirements such as PCI-DSS, and other industry-specific compliance frameworks. Compliance involves implementing appropriate technical controls, maintaining audit trails, and ensuring data handling practices meet legal obligations. Google Cloud provides compliance certifications and attestations that can support regulatory requirements, but organizations remain responsible for their own compliance posture.

Designing the project, dataset, and table architecture to ensure proper data governance

This area addresses structuring Google Cloud projects, datasets, and tables to support effective data governance throughout the data lifecycle. Proper architecture includes organizing resources logically, applying appropriate access controls at each level, implementing labeling and tagging strategies for classification, and establishing clear ownership boundaries. Projects serve as the primary isolation boundary, datasets contain related tables and support access controls, and tables hold the actual data. Governance architecture should consider separation between development and production environments, audit requirements, and the ability to apply organization policies consistently across resources.

Multi-environment use cases (development vs. production)

This area covers designing secure data systems that function appropriately across different environments, particularly distinguishing between development and production use cases. Development environments typically require broader access for testing and debugging, while production environments demand strict access controls, comprehensive logging, and robust security measures. Best practices include using separate projects or folders for each environment, applying organization policies that restrict resource creation in production, implementing different IAM roles and permissions between environments, and ensuring that data used in development is either synthetic or properly de-identified. Environment separation helps prevent accidental changes to production systems and provides a safe space for experimentation.

Key Facts

• Least-privilege access in Google Cloud is implemented using Cloud IAM roles, with predefined and custom roles providing more granular control than basic roles.
• Envelope encryption uses a Data Encryption Key (DEK) to encrypt data and a Key Encryption Key (KEK) managed in Cloud KMS or Cloud HSM to wrap the DEK.
• VPC Service Controls create security perimeters around Google-managed services to mitigate data exfiltration risks.
• The gcp.resourceLocations organization policy constraint restricts the geographic provisioning of new storage and compute resources.
• Format-preserving encryption (FPE) maintains original character lengths to satisfy legacy schema requirements, while cryptographic hashing provides irreversible tokenization for analytical matching.
• BigQuery dynamic data masking enables users to query tables while viewing obfuscated column values, whereas column-level security blocks unauthorized users from querying protected columns entirely.
• Cloud KMS customer-managed encryption keys (CMEK) must reside in the identical geographic region as the data storage and compute resources they protect.