PII Privacy with De-identification and Access Controls

Application of transformational de-identification techniques is the use of Sensitive Data Protection (opens in a new tab) to remove or obscure identifiers while keeping data useful for joins, analytics, and machine learning. Sensitive Data Protection includes Cloud Data Loss Prevention (Cloud DLP) as its API and scans structured data with field-based record transformations and unstructured data with more than 200 built-in classifiers called infoTypes. The transform is applied during inspection and the de-identified output is written as a new copy, so the source remains unchanged and privacy requirements can be met before analytical use.

1.1Reversible Cryptographic Transforms in Sensitive Data Protection

Reversible cryptographic transforms replace sensitive values with tokens that can be de-tokenized back to the original value with the same token encryption key. Format-preserving encryption (FPE) preserves the length and character set of the input, such as keeping a 10-digit numeric value as 10 digits, which maintains compatibility when a legacy application processes the de-identified records. Deterministic encryption (DE) is preferred when strict formatting is not required for legacy compatibility, such as values in a card holder name column, because DE provides a stronger authentication method while still allowing reversal.

1.2Generalization Transforms for Reduced Precision

Generalization transforms are de-identification methods that reduce precision instead of preserving full integrity, including masking, bucketing, date shifting, and replacement. Generalization transforms include pseudonymization and bucketing, where a specific value is replaced with a less unique category that can still be analyzed. For example, bucketing an age value of 27 into an age range of 20-30 preserves the ability to analyze age distribution while reducing the uniqueness that could lead to identification of an individual.

1.3Token Encryption Key Management with Cloud KMS

Token encryption key management controls the cryptographic key required for cryptographic de-identification, where the same key is used to re-identify the original value. The token encryption key must not appear as plaintext in the de-identification template; instead, Cloud Key Management Service (Cloud KMS) creates a wrapped key that the template references. Risk is reduced by using separate token encryption keys for each data element, so compromise of one key does not expose every tokenized field.

Key rotation for token encryption keys breaks token integrity because existing tokens were created with the prior key material. Although the wrapped key can be rotated, rotating the underlying token encryption key requires re-tokenizing the entire dataset to restore consistent joins and re-identification. This tradeoff means rotation planning must balance security hygiene against the cost of a full re-tokenization and coordinated update of downstream BigQuery (opens in a new tab) tables and templates.

1.4Operationalizing De-identification at Scale

Operationalizing de-identification at scale uses templates, pipelines, and query-time functions that separate configuration from execution. Templates decouple transformation configuration from request implementation, enable security control with Identity and Access Management (IAM), and allow the same de-identify and re-identify transformations to be reused across multiple datasets. For ingestion, Dataflow (opens in a new tab) takes data from Cloud Storage, applies Sensitive Data Protection tokenization for data tagged as confidential, and then sends the result to the BigQuery data warehouse, where BigQuery serves as the store for de-identified data and as the backend for an automated re-identification pipeline that can share data through Pub/Sub.

Storage de-identification creates de-identified copies without altering source files through an ordered job flow:

1. Create inspection job: Create a DlpJob inspection job configured to make de-identified copies of the Cloud Storage files.
2. Scan and match: Sensitive Data Protection scans files in the specified location and inspects content according to infoType and transformation configuration.
3. Transform and write: As each file is inspected, data matching sensitive-data criteria is de-identified and written as content to a new file in a designated output bucket.

Common uses for storage de-identification form a set of privacy-pipeline patterns:

• Generate anonymized datasets for non-production environments before staging production files into development, testing, or training.
• Share sanitized file repositories with third parties while preserving original folder hierarchy for partners or auditors.
• Automate privacy pipelines for newly uploaded documents by transforming sensitive data into a designated output bucket without disrupting upstream pipelines.
• Enforce residency and compartmentalization by storing de-identified mirror copies in separate Cloud Storage buckets with distinct IAM access controls.

The same transforms can also be applied at query time, where BigQuery calls remote functions hosted on Cloud Run (opens in a new tab) to encrypt or decrypt values in transit. Query-time de-identification sanitizes real-time query results to minimize access to data not needed for analysis, which can replace a separate transformation pipeline for re-identification or for both de-identification and re-identification across the organization.

Granular access control and governance for personally identifiable information (PII) protection is the combination of least-privilege identity controls, fine-grained data controls inside the warehouse, and centralized oversight of discovery, lineage, and audit. In Google Cloud, Identity and Access Management (IAM) (opens in a new tab) provides the coarse foundation, BigQuery (opens in a new tab) provides column-level security, row-level security, authorized views, and data masking for sensitive attributes, and Knowledge Catalog (opens in a new tab) provides the organization-wide governance plane for lineage, quality, and sensitive-data context.

2.1Role-Based Access Control with Least Privilege

Role-based access control (RBAC) with least privilege means every user, group, or service account receives only the minimum permissions needed to perform its role, such as no access, read-only, or write. This limitation is critical when sensitive data resides in data lakes, feature stores, or model-training hyperparameters, because it prevents data theft, preserves model integrity, and limits the surface area for accidents or attacks. Operational control is separated from data access control, so permission to run a Dataflow job does not by itself grant permission to read the underlying PII.

Granularity in IAM is achieved by mapping product-specific IAM roles to identities based on project needs, access patterns, or resource tags, and by adding IAM Conditions that restrict when a grant applies to data, models, and configurations such as code and hyperparameters. Principal access boundary (PAB) policies further limit what an identity can access even when other policies would allow more. At the storage layer, Cloud Storage enforces application-level granularity through IAM policies on buckets and managed folders, while BigQuery enforces it through IAM roles on datasets and objects inside datasets plus finer row and column controls.

2.2Column-Level Security with Policy Tags

BigQuery column-level security is query-time enforcement that hides entire sensitive columns unless the querier holds the matching policy-tag grant. A taxonomy created in Data Catalog defines policy tags such as high-sensitivity PII, the tag is attached to the column schema, and BigQuery checks the caller's grant at query time before returning that column. BigQuery supports both policy-tag-based control and type-based classification, which allows governance teams to classify once and enforce consistently across many tables.

Attaching a policy tag requires datacatalog.taxonomies.get and bigquery.tables.setCategory. The first permission is included in Data Catalog Policy Tags Admin and Project Viewer, and the second is included in BigQuery Admin (roles/bigquery.admin) and BigQuery Data Owner (roles/bigquery.dataOwner). Viewing taxonomies across all projects in an organization in the console also requires resourcemanager.organizations.get from Organization Viewer, although the API and bq command-line tool do not require it. Only one policy tag can be assigned per column, so a column that contains multiple sensitivities must be modeled or masked accordingly.

Attaching the tag in the console follows an ordered schema-update workflow:

1. Open the table: Locate and select the target table in BigQuery Explorer to open its schema.
2. Start schema edit: Click Edit Schema to make the current schema editable.
3. Select the column: Select the target PII column and click Add policy tag.
4. Choose the tag: Locate and select the policy tag resource name from the taxonomy and click Select.
5. Save the schema: Click Save to apply the updated schema with the attached tag.

The same result can be achieved programmatically by writing the schema with bq show, adding the policy-tag resource name under policyTags.names for the column, and applying it with bq update, or by calling tables.patch for existing tables and tables.insert for new tables. For existing tables, tables.patch is preferred over tables.update because tables.update replaces the entire table resource.

2.3Row-Level Access Policies in BigQuery

BigQuery row-level security is fine-grained filtering that returns only the subset of rows a caller is entitled to see in a table. An access policy defines the row predicate and the authorized identities, and BigQuery applies that filter during query execution so unauthorized rows are never returned. This control restricts visibility horizontally across entities, such as allowing regional analysts to see only rows for their region while denying rows that contain other subjects' PII.

Row-level policies compose with column-level security and IAM dataset permissions. IAM determines whether the identity can query the dataset at all, the row access policy determines which rows survive the filter, and column policy tags determine which sensitive columns remain visible in those surviving rows. Because enforcement happens inside BigQuery at query time, the same physical table can safely serve multiple populations without creating separate PII and non-PII copies.

2.4Authorized Views for Restricted Sharing

An authorized view is a BigQuery view that is explicitly authorized to read source data and then shares only its query result with specific users and groups. Users query the view without receiving access to the underlying source dataset, so the view owner retains control while consumers see only the projected columns, filtered rows, aggregated metrics, or de-identified outputs defined by the view SQL. This boundary makes authorized views the preferred pattern for sharing a PII-reduced subset outside the owning team.

The trust boundary sits between the view and its source tables. The view's service identity holds read access to the source dataset, the consumer holds read access only to the view's dataset, and BigQuery bridges the two only through the authorized-view grant. Data flows from the source tables to the view logic to the authorized consumer, and revoking either the view grant or the consumer's view access immediately stops the flow without changing source-table permissions.

2.5Choosing Among Column, Row, View, and Masking Controls

Choosing among column, row, view, and masking controls means matching the restricted dimension to the privacy risk in the same BigQuery table. The four controls are often layered because they restrict different dimensions of access:

Choose column controls when the risk is the attribute itself, such as Social Security number or diagnosis code. Choose row policies when the risk is the population, such as other tenants or regions. Choose authorized views when sharing across team or organizational boundaries without granting source access. Choose masking when broad analytics can continue on de-identified values, with encryption at rest and in transit remaining on by default and customer-managed encryption keys (CMEK) available through Cloud Key Management Service (Cloud KMS) for stricter key control.

2.6Centralized Governance, Auditing, and Lineage

Centralized governance, auditing, and lineage is the organization-wide plane where Knowledge Catalog centralizes discovery, management, monitoring, and governance of data and AI artifacts across data platforms. Governance flows from ingestion and classification into the catalog, and from the catalog back into enforcement and audit, so policy, lineage, and access stay consistent.

Knowledge Catalog centralizes the following governance functions:

• Data lineage: Track table and column-level lineage, including BigQuery column-level lineage, to trace PII from source to consumption.
• Sensitive-data context: Send Sensitive Data Protection inspection results from BigQuery into the catalog as aspects for continuous PII visibility.
• Data quality and profiles: Manage quality checks, data profiles, and feature metadata for governed tables and feature stores.
• Discovery and vocabulary: Manage discovery across data marts, maintain a business glossary, and enrich metadata with aspects and aspect types.
• Unified and decentralized ownership: Unify governance across BigLake and open formats such as Iceberg and Delta, and support a data mesh where domain teams own their data under central security principles.

Audit and compliance depend on complete, PII-safe telemetry. Cloud Audit Logs collects the audit trail of API calls with a detailed record of user activity and system events, Cloud Logging captures real-time pipeline logs with live tailing, and Cloud Monitoring collects metrics and alerts on log conditions. Audit trails in BigQuery can track data access against sharing agreements and entitlements, while provider usage metrics expose jobs executed, bytes scanned, and subscribers for shared assets. Experiment and pipeline logs must not contain PII or confidential data, and Google-personnel access is separately controlled through Access Approval and recorded in Access Transparency logs.

Automated detection and classification of sensitive data is the use of Sensitive Data Protection (opens in a new tab) and its Cloud Data Loss Prevention API (DLP API) (opens in a new tab) to scan storage systems, identify personally identifiable information (PII), and assign sensitivity and risk levels without manual review. The DLP API provides more than 200 built-in classifiers, called infoTypes, that recognize patterns such as email addresses, phone numbers, credit card numbers, and addresses across Cloud Storage, BigQuery, and relational database imports. Detection operates in two complementary modes: continuous discovery that profiles entire organizations, folders, or projects, and on-demand inspection that pinpoints exact findings inside a specific table or bucket.

3.1Continuous Discovery with Data Profiles

Continuous discovery is an always-on profiling service that is configured at the organization, folder, or project level to find and classify new data as it arrives. When someone creates a new project or uploads a large volume of data, the discovery service automatically scans the new assets and generates data profiles (opens in a new tab) containing metrics and insights, including sensitivity levels and data risk levels. These data profiles act as a source of truth for audit reports and security analysis because they summarize which infoTypes were found, where the data resides, and what protections are missing. Sensitive Data Protection provides a premade multi-page report that breaks down the profiled estate by risk, by infoType, and by location, so owners can see, for example, that low-sensitivity and high-sensitivity data exist in multiple countries.

Data profiles flow from Sensitive Data Protection to governance and security systems that depend on them for monitoring and response. Profiles can be automatically exported to Security Command Center, Google SecOps, and Knowledge Catalog to enrich threat investigation and data management:

• Security Command Center receives profiles to help determine the severity of a vulnerability that might expose sensitive data.
• Google SecOps receives profiles to correlate sensitive-data location with cyber-threat signals.
• Knowledge Catalog receives profiles as aspects to track sensitivity and risk alongside business metadata, lineage, data quality checks, and glossary terms.

After export, downstream teams monitor the profiles for changes that could put data at risk, such as new high-risk columns or loss of access controls, and include the profiles in audit workflows.

3.2Targeted Inspection Across Cloud Storage, BigQuery, and Relational Sources

Targeted inspection is an on-demand scan that identifies the exact locations of PII inside a specific BigQuery table, Cloud Storage bucket, or Vertex AI Agent Platform dataset. An inspection is run as a DlpJob that reads content according to a configured set of infoTypes and finding limits, then returns findings with infoType names, likelihood, and offsets within the scanned content. For relational databases, the common pattern is to use Dataflow with Sensitive Data Protection to tokenize and import data into BigQuery, so inspection and classification travel with the data into the warehouse. For unstructured or freeform columns, inspection is especially important because a high free-text score, especially with multiple infoTypes such as PHONE_NUMBER, US_SOCIAL_SECURITY_NUMBER, and DATE_OF_BIRTH in one comment or notes field, signals hidden PII like customer birth dates in free text.

Discovery depends on broad read access to profile assets, while inspection depends on a focused job configuration for the target asset. In practice, data flows from discovery to inspection: discovery flags a high-risk or high free-text asset, then inspection confirms the precise values before access is restricted or a de-identified copy is created.

3.3Consistent Classification with Built-In, Custom infoTypes and Reusable Templates

Consistent classification with built-in, custom infoTypes and reusable templates is the process of matching content against detectors and labeling each match with a standard name across pipelines. Built-in infoTypes cover common PII and financial identifiers, while custom infoTypes extend detection to organization-specific patterns such as internal account formats, project codes, or domain-specific keywords. A reusable inspection template stores the selected set of built-in and custom infoTypes, likelihood thresholds, and sampling limits in one place, so every ingestion job applies the same detection rules to Cloud Storage uploads, BigQuery loads, and streaming imports.

Consistent templates preserve data utility while reducing compliance risk because analysts, engineers, and auditors share one definition of what counts as sensitive. When templates are versioned centrally and referenced by all DlpJob configurations, new sources inherit the same classifications without copying logic into each pipeline. If a high-risk column is later removed, no new column data profile is generated on reprofiling, so teams that need history for auditing configure the profiler to export profiles to BigQuery before deletion. This separation keeps the detection rule owned by governance and the pipeline owned by engineering, with the template as the boundary between them.

Key Facts

• Format-preserving encryption (FPE) preserves length and character set for legacy compatibility, deterministic encryption (DE) allows reversal with stronger authentication when formatting is not required, and cryptographic hashing cannot be reversed.
• Rotating the underlying token encryption key requires re-tokenizing the entire dataset, and wrapped keys referenced by templates must never expose plaintext in the template.
• BigQuery column-level security denies a tagged column without a grant and allows only one policy tag per column, while row-level security filters rows and authorized views share only the view result without source access.
• Sensitive Data Protection discovery generates organization-wide data profiles with sensitivity and risk levels, while inspection as a DlpJob returns exact finding locations inside a specific table or bucket.
• Data profiles exported to Security Command Center, Google SecOps, and Knowledge Catalog link sensitive-data location to vulnerability severity, threat signals, and governance lineage.