Security and Compliance for Data Workloads

Identity and Access Management (IAM) (opens in a new tab) for data workloads is the system that binds every principal to only the minimum role needed on a specific resource. A principal is a user, group, or service account, a role is a collection of permissions, and access flows from the principal through the role to the dataset, bucket, or project. Organization and project roles control the ability to run jobs or manage projects, while dataset and bucket roles control the ability to read or modify data, and permissions inherit downward unless a lower-level grant narrows them.

1.1Least-Privilege IAM Roles and Inheritance

Least-privilege IAM roles match predefined or custom roles to job functions instead of granting broad ownership. Predefined roles cover common patterns such as BigQuery data viewer or editor, while custom roles combine a user-specified list of permissions for cases such as a pipeline that can read one dataset but cannot list others. Tables inherit permissions from the containing dataset unless table-level access overrides that inheritance, and groups are preferred over individual bindings because membership can change while the binding stays stable. Designs use separation of duties, regular review with policy analysis (opens in a new tab), and IAM Recommender suggestions, and must respect the soft quota of 100 service accounts per project.

1.2BigQuery Table, Column, and Row Controls

BigQuery fine-grained controls decide what a principal sees after dataset access is granted. Table-level access grants a user, group, or service account access to a specific table or view without granting the whole dataset, column-level security restricts sensitive columns through policy tags in a taxonomy, column-level dynamic data masking obscures values while keeping column access, and row-level security filters rows based on a qualifying condition on the user. Policy tags form a hierarchy where 3_Confidential for credit card numbers includes 2_Private and 1_Sensitive, 2_Private for first names includes 1_Sensitive but not 3_Confidential, and 1_Sensitive for credit limit includes neither higher level. Columns without tags remain available to dataset readers, protected columns require the Fine-Grained Reader role on the tag, and the policy still blocks reads after re-identification until access is explicitly granted.

Tenant isolation for large fact tables can use authorized views or tenant-specific subset tables, which differ in scale, performance, and governance:

1.3Context-Aware Access and Perimeter Defense

Conditional and context-aware access adds where, when, and from what device a request is allowed to the question of who the principal is. IAM Conditions enforce attribute-based control, such as allowing a service account to trigger a Vertex AI pipeline only from a trusted Cloud Build trigger, while Identity-Aware Proxy (IAP) policies add IP and device attributes and Access Context Manager with Chrome Enterprise Premium re-evaluates access on every authentication. Session length for Google Cloud services must be set between 1 and 24 hours with a 16-hour default for new customers, and Compute Engine (opens in a new tab) Secure Shell access uses OS Login with 2-Step Verification to block attackers holding only stolen OAuth tokens.

VPC Service Controls mitigates exfiltration by placing Google-managed resources inside a security perimeter and controlling movement across it. Sensitive data sits in its own perimeter, ingress rules control what can enter and egress rules control what can leave, and requests from attacker-controlled endpoints outside the perimeter are blocked even when IAM would allow them. A common pattern uses an internal-data perimeter with BigQuery enforcement, an externally-shared-data perimeter, and a perimeter bridge that allows egress only from internal projects into external-share projects. The service is limited to 10,000 projects inside perimeters organization-wide and 100 perimeters including bridges per organization.

1.4Service Accounts, Impersonation, and Organization Guardrails

Service accounts provide machine identity for pipelines without distributing long-lived keys. A dedicated account owns exactly one pipeline stage, so a training account can read only one Cloud Storage bucket and write only to Model Registry, and Dataflow jobs should pass a dedicated worker account with --serviceAccount instead of using the broad Compute Engine default service account. Secure impersonation avoids key files because an identity holding Service Account User and Service Account Token Creator requests a short-lived token to act as the target account, while Workload Identity Federation extends the same pattern to external identities. Complementary controls include principal access boundary (PAB) policies that cap reachable resources, Privileged Access Manager for just-in-time elevation recorded in audit logs, firewall deny-by-default rules, and organization policies such as disabling external IPs, restricting locations, and blocking service account key creation (opens in a new tab).

Data security for pipelines protects data from ingestion through processing to storage and backup by combining default encryption, envelope encryption, and explicit transit controls. Data flows from on-premises or internet sources through engines such as Dataflow and Dataproc to storage services and backups, and every stage depends on encryption keys that are themselves encrypted and centrally managed. Cloud Key Management Service (Cloud KMS) provides customer control over those keys when compliance requires more than Google-managed defaults.

2.1Envelope Encryption and Default Layers at Rest

Envelope encryption separates the key that encrypts data from the key that protects that key. A data encryption key (DEK) directly encrypts each logical chunk of customer content, while a key encryption key (KEK) wraps the DEK for storage near the data, and KEKs are stored centrally in Keystore. Data moves through an ordered sequence: first incoming data is broken into chunks up to several gigabytes with unique identifiers, then each chunk is encrypted with its own DEK using AES-256, then each DEK is wrapped by a KEK, then access control lists allow only authorized Google services to decrypt, and finally reading requires both the encrypted chunks and the unwrapped DEKs. If a chunk is updated it gets a new DEK, and chunks are replicated in encrypted form so backups never expose plaintext.

Default encryption at rest (opens in a new tab) encrypts all customer content without customer action using multiple redundant layers. At the infrastructure layer, storage-system encryption uses AES-256 except for a small number of Persistent Disks created before 2015 that use AES-128, and at the device layer hard disk drives and solid-state drives apply a second AES-256 encryption with a separate device-level key. Google uses a FIPS 140-2 validated module, certificate 4407, and both AES-256 and AES-128 are recommended by NIST for long-term storage. A chunk may contain data from multiple customers, so logical separation through encryption requires Cloud KMS with customer-managed keys.

2.2Choosing CMEK, HSM, and External Keys

Compliance needs determine whether default encryption is sufficient or customer-managed keys or tokenization are required. Default encryption shifts lifecycle work to Google, while customer-managed options shift responsibility to developers to follow mandated key management that the security team enforces with CMEK organization policies.

Tokenization cannot be centrally enforced and adds cost, latency, and complexity, so it should be avoided when Cloud KMS meets requirements. Customer-supplied encryption keys (CSEK) are not recommended because Cloud EKM supports more services with higher availability, and Cloud KMS Autokey automates provisioning of CMEKs. CMEK-integrated services that encrypt on behalf of the pipeline include Cloud Storage, Cloud SQL, Spanner, BigQuery, Agent Platform, and others, and model artifacts should be stored in Cloud Storage or Model Registry with Cloud KMS encryption.

2.3Encryption in Transit and Hybrid Paths

Encryption in transit protects authenticity, integrity, and privacy while data moves outside Google-controlled physical boundaries. VM-to-VM traffic within a Virtual Private Cloud (VPC) network and between peered VPC networks is integrity-protected and encrypted inside the production network. Traffic from the internet or VPC to Google services terminates at the Google Front End (GFE) for DDoS protection and routing, then uses ALTS for authentication, integrity, and encryption from the GFE to services and between services. Pipelines that accept internet ingress or connect on-premises need more than defaults, including managed certificates on an external Application Load Balancer, Cloud Service Mesh with TLS and mTLS, and HTTPS for all AI and ML API calls. For hybrid paths, security by design (opens in a new tab) uses MACsec over Cloud Interconnect and IPsec over Cloud VPN, with VPC Service Controls perimeters and deny-by-default firewall rules to prevent exfiltration.

Privacy protection for personally identifiable information (PII) is the use of Sensitive Data Protection to discover, classify, de-identify, and govern sensitive data while keeping it useful for analytics. Sensitive Data Protection includes Cloud Data Loss Prevention (Cloud DLP) as its API and scans structured data with field-based record transformations and unstructured data with more than 200 built-in classifiers called infoTypes. The transform is applied during inspection and the de-identified output is written as a new copy, so the source remains unchanged before analytical use.

3.1Reversible and Irreversible De-identification Transforms

Reversible cryptographic transforms replace sensitive values with tokens that can be de-tokenized with the same token encryption key. Format-preserving encryption (FPE) preserves length and character set, such as keeping a 10-digit value as 10 digits for legacy applications, while deterministic encryption (DE) is preferred when strict formatting is not required because it provides stronger authentication while still allowing reversal.

Generalization includes pseudonymization and bucketing, where a specific value such as age 27 becomes a range such as 20-30 to preserve distribution while reducing uniqueness. The token encryption key must not appear as plaintext in the template; Cloud Key Management Service (Cloud KMS) creates a wrapped key that the template references, with separate keys per data element recommended. Rotating the underlying key breaks token integrity and requires re-tokenizing the entire dataset to restore joins and re-identification.

3.2Discovery Profiles Versus Targeted Inspection

Automated detection operates in two complementary modes: continuous discovery that profiles organizations, folders, or projects, and on-demand inspection that pinpoints exact findings inside a specific table or bucket. Discovery automatically scans new assets and generates data profiles with sensitivity levels, data risk levels, and free-text scores, which act as a source of truth for audit and security analysis. Inspection runs as a DlpJob that reads content by infoTypes and returns findings with infoType names, likelihood, and offsets, which is critical for unstructured columns where multiple infoTypes in one notes field signal hidden PII.

Data flows from discovery to inspection in practice: discovery flags a high-risk asset, then inspection confirms precise values before access is restricted. Profiles can be exported to Security Command Center for vulnerability severity, Google SecOps for threat correlation, and Knowledge Catalog as aspects for lineage and governance. Consistent classification uses built-in infoTypes for common identifiers, custom infoTypes for organization-specific patterns, and reusable templates that store detectors and thresholds so every pipeline applies the same rules.

3.3Column, Row, View, and Masking Controls for PII

Granular warehouse controls enforce least privilege inside BigQuery after IAM grants dataset access. Column-level security hides entire tagged columns at query time unless the caller holds the tag grant, row-level security filters rows by an access policy during execution, authorized views share only the view result without source access, and data masking obscures sensitive values while returning the row. Attaching a policy tag requires datacatalog.taxonomies.get and bigquery.tables.setCategory, only one tag can be assigned per column, and for existing tables tables.patch is preferred over tables.update because update replaces the entire resource. The trust boundary for an authorized view sits between the view and its source: the view identity holds source read access, the consumer holds only view read access, and revoking either stops the flow.

3.4Governance, Lineage, and Audit Telemetry

Centralized governance ties discovery, enforcement, and audit together through Knowledge Catalog. Knowledge Catalog centralizes data lineage including BigQuery column-level lineage, Sensitive Data Protection results as aspects, data quality checks and profiles, discovery with business glossaries, and unified governance across BigLake and open formats such as Iceberg and Delta with data-mesh ownership. Operational de-identification separates configuration from execution with templates, Dataflow pipelines from Cloud Storage to BigQuery, storage jobs that create de-identified copies in a designated output bucket while preserving hierarchy, and BigQuery remote functions on Cloud Run for query-time encryption. Audit telemetry uses Cloud Audit Logs for API activity, Cloud Logging for real-time pipeline logs, and Cloud Monitoring for metrics and alerts, while experiment logs must not contain PII and Google-personnel access is controlled by Access Approval and recorded in Access Transparency logs. Under HIPAA practices (opens in a new tab), teams sign a Business Associate Agreement, avoid pre-GA services for protected health information, and never place sensitive data in metadata or log fields.

Enforcing data sovereignty with Google Cloud Organization Policies means using centralized guardrails to allow storage and processing only inside approved countries or regions. The central control is the resource location restriction (constraints/gcp.resourceLocations), which defines an allowedValues list of permitted multi-regions, regions, and zones. When attached to an organization, folder, or project, any request to create a location-based resource outside that list is denied before data can be stored there.

4.1Resource Location Constraint and Policy Inheritance

The resource location constraint blocks out-of-boundary provisioning only for services that support resource locations, so administrators must confirm each planned service from BigQuery to Cloud Storage to Compute Engine can be pinned to the required region. An administrator sets allowedValues such as in:us-locations plus United States regions for a United States boundary or europe-west1 and europe-west4 for a European Union boundary, and provisioning fails when policy and requested location are incompatible. Locality planning must answer where data rests, whether encryption mitigates locality, whether processing may leave the rest location, whether multiple regions are allowed, and whether access must be restricted by identity or network. Organization policy enforcement flows from the Organization resource down through folders to projects, so one boundary covers every child workload, and Assured Workloads automates this by creating a folder with a control package that sets location and control-plane boundaries.

4.2Single-Region, Dual-Region, and Multi-Region Storage Choices

Storage topology is the choice of where data at rest lives and how many separate areas hold copies. A single-region topology keeps all copies inside one region for strict sovereignty, while dual-region and multi-region topologies store data in at least two separate areas for higher availability. Cloud Storage objects in dual- or multi-region buckets are stored in at least two areas regardless of storage class, with asynchronous synchronization between areas.

If limited to a single region, cross-geographic redundancy requires a multi-cloud or hybrid second copy in the same area or two single-region buckets with dual-write or object rewrite. BigQuery dataset locations are either regional or multi-regional, store copies in two zones, and cannot be changed after creation, so strict workloads must start regional. Cloud Spanner placement is controlled by instance configuration: regional configurations keep one read-write replica per zone in one region, while multi-region configurations replicate across regions for regional survival at the cost of broader residency scope, and Spanner can move between configurations without downtime.

4.3Regional Processing, Egress Perimeters, and Dataflow Key Rules

Governing regional processing, egress, and key residency keeps storage, processing, and key operations inside a defined jurisdiction. Data residency (opens in a new tab) commitments apply to Customer Data at rest and in use for location-configurable services, where processing stays in the country of the selected region or in the countries of a multi-region, but Customer Data excludes identifiers and labels and Service Data follows separate geography rules. VPC Service Controls creates a context-based perimeter that controls data movement and overrides permissive IAM settings, so even a misconfigured public grant does not allow outside access. A governed flow runs in order: first upload to BigQuery or Cloud Storage with CMEK and Autokey, then encrypt with Cloud KMS, then classify with Sensitive Data Protection, then permit entry through Access Context Manager levels, and finally deny by default to anyone not specified.

Localized keys keep cryptographic operations in-jurisdiction through Cloud KMS, Cloud External Key Manager, Cloud HSM with FIPS 140 validation, and Key Access Justifications with Access Transparency and Access Approval to approve or deny Google access. Dataflow enforces key residency by supporting only regional Cloud KMS keys that match the job region, so the key ring and symmetric key must be created in the same region as the job and global or multi-regional keys do not work. For batch jobs all data including data keys is protected by CMEK, while streaming jobs created after March 7, 2024 protect all user data and earlier streaming jobs do not protect data keys for windowing and grouping until drained and restarted. Protected state includes worker Persistent Disks, Shuffle state, Streaming Engine state, and Cloud Storage buckets for temporary, code, and sampled data, while job metadata such as names, parameters, IDs, and worker IPs is not encrypted with CMEK. Assured Workloads (opens in a new tab) packages these controls for regulated boundaries with in-scope services that support CMEK, VPC Service Controls, and transparency logs.

Legal and regulatory compliance ties together cryptographic ownership, privacy transformation, and residency control so regulated data can be decrypted only by approved parties, seen only in approved form, and stored only in approved places. In Google Cloud, Cloud Key Management Service (Cloud KMS) centralizes key creation, rotation, destruction, and access control, while Customer-Managed Encryption Keys (CMEK) let supported services encrypt data with customer-controlled keys. Combined with organization policies, usage tracking, and audit logging, this model supports sovereignty, crypto-shredding, and proof of compliance.

5.1CMEK Configuration and Crypto-Shredding Control

Data-at-rest protection starts with Google default encryption using FIPS 140 validated AES-256, with envelope encryption linking customer control to that default. Each service generates a data encryption key (DEK) to encrypt data, then wraps the DEK with a key encryption key (KEK) in Cloud KMS, and CMEK services use the customer key as the KEK so revoking it removes the ability to unwrap DEKs. Configuring CMEK follows an ordered dependency: first create a key ring geographically near the resources in the same or a separate key-management project, then grant roles/cloudkms.cryptoKeyEncrypterDecrypter to the service agent, then attach the key at resource creation in the same location, and finally allow requesters to access data through the service without direct key access. If the service-agent role is revoked or the CMEK is disabled or destroyed, data becomes inaccessible, which enables crypto-shredding but can cause permanent loss if the key stays unavailable. Google security overview (opens in a new tab) principles require that decrypted key material cannot be exported and Google personnel cannot access unencrypted customer material.

5.2Key Protection Levels and External Options

Cloud KMS protection levels define where key material lives and what assurance it provides for availability, rotation, quota, and FIPS validation. Software keys use FIPS 140-3 Level 1 modules with automatic rotation for symmetric keys, hardware keys use FIPS 140-2 Level 3 HSMs, and Cloud External Key Manager (Cloud EKM) keeps keys outside Google Cloud, although some CMEK services do not support EKM keys. Options include software keys in all locations, imported software keys with manual rotation by new versions, multi-tenant Cloud HSM keys in Google-managed clusters, single-tenant Cloud HSM keys in dedicated partitions with quorum and two-factor administration, and external EKM keys where the customer owns availability. Customer-supplied encryption keys (CSEK), where the customer supplies material per operation and Google stores nothing, lack centralized rotation and audit through Cloud KMS, and Compute Engine CSEK is deprecated in favor of CMEK.

5.3Privacy-Preserving Ingestion and Residency Enforcement

A secure governed pattern combines CMEK, automated classification, and a network perimeter that overrides permissive IAM. Data flows from the uploader to BigQuery or Cloud Storage encrypted with CMEK from Cloud KMS Autokey, classification flows from Sensitive Data Protection to the stored data for masking, and access decisions flow from Access Context Manager to the VPC Service Controls perimeter that denies anyone not in the access level. Ingestion-time de-identification flows from Cloud Storage or Pub/Sub through Dataflow calling Sensitive Data Protection to BigQuery warehouses split into confidential and non-confidential projects, with separate ingestion, wrapped de-identification, and warehouse CMEKs. Query-time de-identification instead calls Cloud Run remote functions such as fns.dlp_freetext_encrypt() from BigQuery without maintaining a separate copy, while storage de-identification creates DlpJob copies in a designated output bucket preserving hierarchy. Column protection adds Tink client-side encryption compatible with BigQuery AEAD functions, where a DEK encrypts the column and a KEK wraps the DEK with encrypted data and wrapped keys in separate tables, plus policy tags where 3_Confidential grants all lower levels, 2_Private grants 2_Private and 1_Sensitive, and 1_Sensitive grants only itself. Residency enforcement uses the Organization Policy Service resource locations constraint to block new resources in denied locations without moving existing data, requires BigQuery federated sources colocated with the dataset, pins Cloud Storage buckets to single, dual, or multi-region tradeoffs, and uses Assured Workloads frameworks (opens in a new tab) with external keys and perimeters where required.

Designing project, dataset, and table architecture for governance means separating storage, access, and key controls so confidential data stays isolated and auditable. BigQuery datasets are child resources of projects in the IAM hierarchy, and a table inherits permissions from its dataset unless table-level access breaks that inheritance. The design spans projects for trust boundaries, datasets for permission boundaries, and tables, columns, and views for fine-grained enforcement with lineage and audit.

6.1Project Separation by Sensitivity and Environment

Project segregation places datasets with different sensitivity in different projects so IAM, encryption, and perimeter rules can differ by project. The confidential project holds re-identified warehouses with restricted columns and a separate CMEK, the non-confidential or shared project holds de-identified copies for broader use, the ingestion project holds untrusted incoming data and de-identification pipelines, and the governance project in the common folder holds keys, taxonomies, catalog metadata, and logging separate from data. Data flows from the source into Cloud Storage or Pub/Sub, then through Dataflow into the warehouse inside the perimeter, with Cloud VPN or Cloud Interconnect for external movement and allowlisted IPs for expected environments.

Google Cloud Organization sits at the top, folders group bootstrap, common, production, non-production, and development in the middle, and projects hold datasets at the bottom, with most warehouses in production and governance in common.

6.2Dataset Inheritance and Table-Level Isolation

IAM hierarchy grants roles on a specific resource, where organization and project roles affect job execution and management while dataset roles affect data access inside the project. Basic roles such as Owner, Editor, and Viewer apply broadly and should be avoided for sensitive data, predefined roles support patterns such as data viewer, data editor, job user, and Fine-Grained Reader, and custom roles combine a user-specified permission list when no predefined role fits. Table-level access breaks dataset inheritance by naming users, groups, and service accounts that can read a specific table or view, which supports designs where a storage project holds datasets, a separate project runs jobs, and each sensitive table carries its own grant. Guardrails from the Organization Policy Service inherit downward, including disabling service account creation, blocking key creation and automatic default grants, requiring OS Login, restricting protocol forwarding and Shared VPC subnetworks, requiring CMEK for BigQuery, and restricting Cloud Run ingress. Prevention of exfiltration uses VPC Service Controls perimeters for ingestion, confidential data, external ingestion, and governance, with explicit ingress, egress, or bridge rules allowing movement such as from internal to external-share projects.

6.3Policy Tags, Masking, and Tenant Isolation Patterns

Column-level governance restricts confidential columns even when the table is readable, where policy tags bound to columns require the Fine-Grained Reader role plus table read access and untagged columns remain broadly readable. The reference taxonomy grants downward only: 1_Sensitive for credit limit grants only that level, 2_Private for first name grants that plus 1_Sensitive, and 3_Confidential for credit card numbers grants all three levels, and controls persist after re-identification. Dynamic data masking obscures values for groups while allowing queries, row-level security filters rows by conditions such as SESSION_USER() matching tenant or region, and AEAD column encryption with the Tink library stores encrypted data and wrapped DEKs in separate tables so decryption needs both table and KMS access. Tenant isolation on BigQuery (opens in a new tab) can use authorized views or subset tables: views inherit base partitioning and must stay in the base region with a 2500 authorized-resource limit per dataset and automatic column-policy inheritance, while subset tables allow per-tenant partitioning, regionalization, and isolated logging at the cost of copying and reapplying policies. Centralized lineage and audit use Knowledge Catalog for column lineage, quality, discovery, and BigLake and open-format governance, Admin Activity logs for configuration changes, Data Access logs for BigQuery reads, and regional storage with lifecycle controls, as described in the secured warehouse blueprint (opens in a new tab).

Multi-environment design applies differentiated identity, network, and data controls to development, testing, and production based on purpose and sensitivity. Production protects live confidential data with the strictest boundaries and narrowest access, while development and testing favor speed on minimized, de-identified, or synthetic data. This follows secure by design, which embeds security throughout the lifecycle, and secure by default, which starts every environment from secure settings.

7.1Differentiated Security Postures for Development and Production

A differentiated posture limits blast radius so compromise in one environment cannot spread to another, using defense-in-depth, documented requirements, simplification for recovery, and risk assessment under shared responsibility. Production isolates sensitive data in its own service perimeter, enforces multifactor authentication, and denies traffic and privilege by default, while development minimizes PII collection and testing mirrors production controls on classified, tagged, and tokenized copies rather than raw data. Role-based access with least privilege gives every user only no-access, read-only, or write as needed, with tighter separation for production lakes, feature stores, and hyperparameters. Granularity comes from mapping product roles by project, tags, or patterns, adding IAM Conditions for attribute-based limits, and applying bucket and managed-folder policies in Cloud Storage plus dataset, row, and column controls in BigQuery. Service accounts are dedicated per pipeline and environment, such as a training account that reads only one bucket and writes only to Model Registry, with IAM Conditions such as trusted Cloud Build origin and organization guardrails that disable external IPs, restrict locations, and block key creation in production. Context-aware elevation flows from low-privilege base identity to temporary privilege only after checks pass, using Privileged Access Manager (opens in a new tab) for just-in-time elevation, principal access boundaries to cap reach, and IAP with Access Context Manager and Chrome Enterprise Premium for IP and device checks.

7.2Artifact Promotion with Scans and Attestations

Repeatable promotion moves the same verified artifacts from development through test to production with security gates at each step. Cloud Build creates and tests artifacts, Artifact Registry stores them, Artifact Analysis scans them, Cloud Deploy promotes them in sequence, and Binary Authorization blocks deployment lacking approvals. The ordered workflow is: first commit triggers Cloud Build to compile and test, then passing code is packaged once such as a Beam JAR or template plus JAR, then artifacts are staged in the deployment project, then they deploy to preproduction for end-to-end tests without rebuilding, and finally production updates only after those checks succeed. Cloud Deploy automates delivery to GKE, GKE Enterprise, and Cloud Run with approvals, rollbacks, and audit, while Binary Authorization checks attestations proving the image came from a specific build system, complies with vulnerability policy, or meets promotion criteria, and denies deployment when attestations are missing. Continuous scanning extracts package information at upload to Artifact Registry, monitors metadata against new vulnerability sources, and updates or deletes occurrences, complemented by SLSA source and provenance checks, Web Security Scanner, Assured Open Source Software, and metrics on remediation and scan pass rates. Hardening covers ephemeral isolated builds with SLSA Level 3 provenance, Registry remote and virtual repositories to reduce dependency confusion, SBOM and VEX transparency, and Terraform with Open Policy Agent for conformance. Data pipelines follow the same artifact-forward rule: batch launches a new Dataflow job directly, via Airflow, or via Cloud Scheduler, while streaming replaces or updates the job or runs parallel jobs during transition to avoid disruption.

7.3Data Minimization and Consistent Governance Across Environments

Consistent governance applies the same classification, encryption, and lifecycle rules everywhere while varying how much sensitive data each environment sees. Knowledge Catalog centrally discovers and governs data and AI artifacts across BigQuery, lakes, and open tables with glossaries, aspects, quality checks, profiles, column lineage, and BigLake and Iceberg and Delta unification, while Sensitive Data Protection results flow into the catalog so one production classification propagates to lower environments. Encryption uses default Google-managed keys unless compliance requires CMEK in Cloud KMS with HSM or external keys, or application-layer tokenization before persisting to BigQuery for analysts who must not see raw fields. Separation keeps raw values in production under row, column, dataset, and service-account controls while development uses pseudonymized, de-identified, generalized, tokenized, or synthetic substitutes, with BigQuery sharing and clean rooms for cross-boundary analysis without copying raw tables. Lifecycle governance runs in order: first categorize with Sensitive Data Protection, then de-identify with masking or encryption, then store inside a VPC Service Controls perimeter with encrypted values and wrapped keys in separate tables, and finally re-identify on demand with logging. Lineage from Vertex AI Pipelines and ML Metadata flows into the catalog, Model Registry versions models, and auditability uses Audit Logs, Logging, and Monitoring without logging PII, with Assured Workloads boundaries for FedRAMP and CJIS and security overview controls (opens in a new tab) blocking unapproved movement from production to development except through de-identification paths.

Key Facts

• BigQuery column policies use hierarchical tags where 3_Confidential includes 2_Private and 1_Sensitive, protected columns require the Fine-Grained Reader role, and base-table policies apply automatically to authorized views but must be reapplied to each subset table.
• Envelope encryption encrypts each chunk with its own data encryption key using AES-256 and wraps each DEK with a key encryption key in Cloud KMS, so disabling or destroying the CMEK makes the data unreadable.
• VPC Service Controls creates a service perimeter that controls data movement across its boundary and blocks exfiltration even when IAM would allow access.
• The constraints/gcp.resourceLocations organization policy denies creation of location-based resources outside approved regions, while BigQuery dataset locations cannot be changed after creation and Dataflow requires regional Cloud KMS keys matching the job region.
• The same verified artifact moves from development through preproduction to production without rebuilding, with Artifact Analysis scanning at upload and Binary Authorization blocking deployment without required attestations.