Sensitive Data Protection Discovery and Pseudonymization

Non-cryptographic de-identification in Sensitive Data Protection (SDP) (opens in a new tab) removes or generalizes sensitive values without encryption keys. SDP can inspect data with more than 100 built-in classifiers, then apply masking, redaction, bucketing, date shifting, and tokenization to reduce handling risk. Because masking, redaction, bucketing, and date shifting are generally one-way, the sanitized replica cannot be reversed to recover original values. Choose these methods when sanitized data must flow to storage, export, or analytics and no authorized workflow needs the originals.

1.1Non-Cryptographic Transformation Choices

Non-cryptographic transformations in SDP protect sensitive records by replacing, removing, or generalizing detected values while leaving non-sensitive content usable. Masking obscures characters in a detected value, redaction removes matched text or an image region and replaces it with a placeholder or opaque rectangle, and bucketing and date shifting generalize values to reduce identifiability. These methods keep raw identifiers from traveling to downstream storage or analytics, which lowers risk for large volumes of personally identifiable information (PII). Choose a non-cryptographic transformation when the pipeline must sanitize data before storage, export, or analytics and reversal is not required. If high-risk data is not needed at all, removal is preferred over transformation, and deleted BigQuery columns will no longer produce column data profiles on reprofiling.

1.2Structured and Unstructured Transformation Scopes

SDP divides de-identification work into two transformation scopes that control how inspection findings flow into protection. recordTransformations apply a transformation configuration per field and are designed for structured or tabular data, while infoTypeTransformations inspect for sensitive data and transform each finding and are designed for unstructured or free-text data.

A per-field rule protects an entire column uniformly, while an infoType-based rule preserves surrounding narrative and changes only the detected span. Both scopes can support custom infoTypes in addition to built-in detectors.

1.3Text Classification and Automatic Text Redaction

Text classification in SDP identifies sensitive entities in free-form input without changing the original text. Classification of free-form input returns findings organized by InfoType, Likelihood, and Offset, where offset marks where the potential match was found in the string. Common pipeline uses include sanitizing user input before database persistence, stripping PII from chat histories and support tickets before review or training ingestion, and redacting proprietary data from generative AI prompts before content flows to external models. Automatic text redaction takes the same detection result and produces sanitized output, for example replacing an email address, National Provider Identifier, and driver's license with *** while labels and sentence structure remain. The application flow is fixed: data flows from the input handler to SDP for inspection and redaction, then only the redacted text flows to the database, log, or model.

1.4Image Redaction for Text and Sensitive Objects

Image redaction in SDP detects sensitive text and objects within an image and returns a redacted copy where each detection is obscured by an opaque rectangle. Text detection uses object character recognition (OCR) and can redact text matching all default infoTypes, only the infoTypes listed in the image redaction configuration, or all detected text. Object detection redacts regions that typically contain sensitive information, such as OBJECT_TYPE/BARCODE, OBJECT_TYPE/LICENSE_PLATE, and OBJECT_TYPE/WHITEBOARD. By default SDP uses black rectangles, but the configuration can assign a different color to each infoType. Image redaction has strict input boundaries: SDP can redact many image types, including JPEG, BMP, and PNG, while content redaction is not supported for SVG, PDF, XLSX, PPTX, or DOCX files. An image redaction request cannot include limits in its inspection configuration because limits can cause only some data to be redacted; if limits is set, SDP generates an error.

1.5Automated Pipelines With Redaction Templates

Automated de-identification pipelines use SDP templates and managed keys to sanitize data at scale while limiting who can see the protection logic. A reference architecture separates three ownership zones: a Dataflow streaming de-identification pipeline that transforms sensitive text, configuration management for SDP templates and keys accessible only to a small group such as security admins, and a validation and re-identification pipeline that checks de-identified copies and re-identifies data at scale when authorized. Data flows from source storage through the de-identification pipeline to a de-identified replica, so downstream analytics depends on the sanitized copy rather than the raw dataset. Restricting template and key access preserves the trust boundary between pipeline operators and security admins. Cloud Data Fusion (opens in a new tab) integrates the same controls through an SDP plugin with three transforms:

• PII Filter transform separates sensitive records from non-sensitive records based on an inspection template, such as credit card or Social Security number detectors.
• Redact transform identifies sensitive records and applies masking or encryption transformations defined by predefined filters or a custom template.
• Decrypt transform decrypts records previously encrypted with a reversible method, and it depends on the same configuration used by the Redact transform that encrypted the data.

The PII Filter transform can filter on records or fields, and filtering by field avoids pipeline failure when a record exceeds 0.5 MB under SDP content limits. Its error handling defines the consequence of a sensitive-data error: stop the pipeline as soon as an error is encountered, skip the record without reporting an error and continue, or send errors to the error port and continue. The Redact transform cannot combine predefined filters and a custom template; the pipeline must use either matching rules grouped by category or a custom template ID.

1.6Statistical De-Identification With K-Anonymity

Statistical de-identification in SDP measures whether quasi-identifiers in a dataset could be used to re-identify individuals after masking or generalization. K-anonymity is a property where the quasi-identifiers for each person are identical to at least k minus 1 other people in the same dataset, meaning each released record blends into a group of at least k records. Computing k-anonymity lets a data owner test whether bucketing, masking, or other generalizations have created large enough equivalence groups before export or analysis. Choose stronger generalization or suppression when the computed k value is too low for the intended sharing boundary.

Sensitive Data Protection (SDP) (opens in a new tab) is the service for discovering, classifying, and protecting sensitive data such as PII. Cloud Data Loss Prevention (Cloud DLP) is now a part of Sensitive Data Protection, while the API name remains the Cloud Data Loss Prevention API (DLP API). SDP scans data where it is stored, so data does not leave its region of origin during inspection or profiling. Discovery profiles show where risk exists across stored data, while detailed inspection proves exactly what and where the sensitive values are.

2.1Inspection Templates as Reusable Detection Configurations

Inspection templates in SDP are reusable configurations that persist what to look for and how to look for it. A template stores the selected infoTypes, optional rulesets, and a confidence threshold, and it can then be referenced by inspection jobs and discovery scan configurations. A template is valid only when at least one infoType is selected, and the same template can be reused across Cloud Storage, BigQuery, and Firestore in Datastore mode (Datastore). Templates are regional resources with strict data residency behavior: a template used for profiling must be in the same region as the data to be profiled, and SDP never applies a template outside the region where that template is stored. A new template can be created in the global region for broad coverage, or dedicated templates can be supplied per region by full resource name in the form projects/PROJECT_ID/locations/REGION/inspectTemplates/TEMPLATE_ID.

2.2Built-In and Custom InfoTypes for Classification

InfoTypes in SDP are the classifiers that define what counts as sensitive data in an inspection template. Built-in infoTypes cover more than 100 common sensitive categories, such as PERSON_NAME, PHONE_NUMBER, US_SOCIAL_SECURITY_NUMBER, and DATE_OF_BIRTH, and are managed through Manage infoTypes in the console. Built-in detectors combine pattern matching with context signals, which is why tuning is often needed to control false positives. Custom infoTypes extend detection to organization-specific sensitive data, including regular expressions, dictionaries and word lists, and stored infoTypes. Custom infoTypes are created and managed alongside built-in infoTypes in the same inspection template workflow. Use custom infoTypes when built-in detectors do not cover proprietary identifiers, internal codes, or domain-specific terms that still create data risk.

2.3Tuning Detection Accuracy and False Positives

Detection tuning in SDP controls the tradeoff between missing sensitive data and generating false positives. The confidence threshold sets the minimum likelihood a match must reach before SDP reports it as a finding, so raising the threshold reduces noise but can miss weaker matches. Rulesets modify detector behavior for specific infoTypes, including hotword rules that raise likelihood when a nearby word provides context. For example, a hotword rule can increase the likelihood of a PERSON_NAME match when the word patient appears nearby, which focuses the detector on patient names in a medical database rather than all person names. A separate exclusion behavior can omit overlapping matches, such as omitting a DOMAIN_NAME match that is part of an email address, so only the more specific finding is kept. Evaluate tuning by exporting sample findings and checking the quote, matched infoType, and location against what was intended to be flagged.

2.4Inspection Jobs, Job Triggers, and Synchronous Inspection

Inspection jobs in SDP perform detailed, on-demand scans of specific Cloud Storage buckets, BigQuery tables, or Datastore kinds and return individual findings. Job triggers automate inspection jobs on a schedule so repositories are rescanned as data changes. The trigger defines the input data, references an inspection template for detection parameters, adds post-scan actions, and then runs on its schedule. Use an inspection job when investigation needs exact locations and quotes, such as after discovery reports a high free-text score or multiple infoTypes in a BigQuery column or Cloud Storage bucket. For data that has not yet reached backend storage, the synchronous content methods of the DLP API inspect data in near real time to proactively prevent sensitive data from entering storage.

2.5Organization-Level Discovery Profiles and Scan Configuration

Discovery profiles in SDP are continuously generated summaries that describe sensitivity and data risk across an organization, folder, or project. The discovery service automatically finds new projects, tables, and buckets as the organization grows and reports metrics for audit and security workflows. Discovery acts as the source of truth for where sensitive data lives, and a premade report can break results down by risk, infoType, and location. To use an existing inspection template in a discovery scan configuration, the full resource name must be supplied so the Region field populates correctly:

1. Open template list: Go to the inspection templates list and select the project that contains the template.
2. Open template detail: Select Configuration > Templates > Inspect, then click the template ID.
3. Copy full name: Copy the full resource name in the form projects/PROJECT_ID/locations/REGION/inspectTemplates/TEMPLATE_ID.
4. Paste into scan configuration: On the Create scan configuration page, paste the name into the Template name field.
5. Cover remaining regions: Click Add inspection template for each additional region, and optionally add a global template to cover regions without a dedicated template.

Discovery scan configurations also define a catch-all schedule for resources that do not match any custom schedule. To stop profiling unmatched resources, turn off Profile the resources that don't match any custom schedule.

2.6Evaluating Findings and Taking Remediation Action

Scan findings in SDP connect a detected quote to its infoType, location, and the calculated sensitivity and data risk of the containing resource. Sample discovery findings represent a subset of all matches, normally around 10 per bucket, and each finding includes the actual detected string and its exact location. Sample findings do not necessarily represent every infoType discovered, so they are best used to validate whether the inspection configuration matches the intended sensitive data. Discovery scan configurations can automatically send profile insights to other Google Cloud services after a table or bucket is profiled:

• Publish to Security Command Center: Sends profile findings to Security Command Center for triage and response planning, which requires organization-level activation.
• Publish to Google Security Operations: Adds profile metrics as context for security investigations, such as whether an accessed resource holds high-sensitivity data.
• Save data profile copies to BigQuery: Keeps a history of profiles in a BigQuery table for audits and single-view analysis across regions, defaulting to dataset sensitive_data_protection_discovery and table discovery_profiles when no table is supplied.
• Save sample discovery findings to BigQuery: Saves quotes, matched infoTypes, locations, and sensitivity levels to a separate BigQuery table for tuning queries and joins against profile copies.
• Publish to Pub/Sub: Emits notifications for programmatic workflows that catch and remediate high-sensitivity or high-risk results.

High data risk means sensitive data was found without additional protections, while a high free-text score means an unstructured column such as a notes field likely contains multiple PII values. For BigQuery (opens in a new tab) columns with sensitive data, apply policy tags or column-level controls, de-identify raw values with masking or tokenization, or remove unneeded high-risk data. For free-text fields in BigQuery and Cloud Storage, run an on-demand inspection to locate exact PII before applying de-identification.

Cryptographic pseudonymization in Sensitive Data Protection (SDP) (opens in a new tab) replaces raw PII with tokens that preserve analytical utility while reducing exposure risk. Tokenization, also called pseudonymization or surrogate replacement, obfuscates sensitive identifiers so de-identified replicas can still be joined, aggregated, or analyzed without handling original values. SDP performs these transformations at scale through de-identification templates executed in automated pipelines, with re-identification strictly controlled by keys and configuration.

3.1Cryptographic Pseudonymization Options

Cryptographic pseudonymization options in SDP use a cryptographic key to produce reversible or one-way tokens that stand in for the original PII. Format-Preserving Encryption (FPE) encrypts a value while keeping its original format and length, so a tokenized phone number or identifier still passes format checks in downstream systems. Deterministic encryption always produces the same token for the same input value when the same key and configuration are used, which preserves referential integrity across rows, tables, and datasets for joins and counts. Cryptographic hashing applies a one-way function with a key to produce a token that cannot be decrypted back to the original, which removes re-identification risk but also removes recovery. Choose FPE when downstream systems require the original format, deterministic encryption when joins and counts must remain consistent, and hashing when no authorized reversal should exist.

3.2Cloud KMS-Wrapped Keys and Surrogate InfoTypes

Cloud KMS-wrapped keys in SDP protect the data-encryption key that actually tokenizes the values. The transformation configuration contains a kmsWrapped block with a wrappedKey, which is the encrypted data key, and a cryptoKeyName in the form projects/PROJECT_ID/locations/global/keyRings/dlp-keyring/cryptoKeys/dlp-key that points to the Cloud Key Management Service (Cloud KMS) key that unwraps it. At transformation time SDP calls Cloud KMS to unwrap the key, performs encryption or decryption in memory, and writes only the token to the output, so the pipeline depends on both SDP and Cloud KMS being available and authorized. Google recommends wrapped keys for actual sensitive workloads, while unwrapped keys are suitable only for demonstration because anyone with the configuration can reverse the tokens. The surrogate infoType labels the token so SDP can find it later for re-identification. For example, a deterministic configuration for EMAIL_ADDRESS specifies a surrogateInfoType such as EMAIL_ADDRESS_TOKEN, and the output text keeps its surrounding structure while the email itself is replaced by its token. Access to the de-identification template and the Cloud KMS key must be limited to a small group such as security administrators, because possession of the method plus the key equals ability to re-identify. This creates a trust boundary where analysts and downstream environments on one side see only tokens, while key custodians on the other side retain the means to restore originals.

3.3Reversible Versus Irreversible Tokenization Tradeoffs

Reversible pseudonymization in SDP preserves the ability to restore originals under control, while irreversible tokenization permanently removes that path to minimize handling risk. Reversible methods require long-term key management, strict access separation, and a defined re-identification workflow; irreversible methods require no key custody for recovery but cannot support authorized reversal, debugging, or disclosure workflows.

3.4Applying Cryptographic Transformations in Pipelines

SDP applies cryptographic transformations through two transformation scopes that match the shape of the data. The recordTransformations scope applies a transformation per field in structured or tabular data, such as tokenizing every value in a US_SOCIAL_SECURITY_NUMBER column while leaving other columns unchanged. The infoTypeTransformations scope inspects content for findings and transforms only the finding, so in unstructured free text SDP can locate an SSN inside a sentence and encrypt just that value while leaving the rest of the sentence intact. The transformations execute inside managed pipelines that separate duties and preserve scale. Source data flows from storage into a Dataflow streaming de-identification pipeline that applies the SDP template and writes a de-identified replica, and a separate validation and re-identification pipeline later reads tokenized copies and calls re-identify to restore values at scale when authorized. In Cloud Data Fusion (opens in a new tab), the same model appears as three plugin transforms: PII Filter separates sensitive records from non-sensitive records without changing values, Redact identifies sensitive records and applies the configured masking or encryption transformation, and Decrypt restores records previously encrypted with a reversible algorithm using the identical Redact configuration and wrapped key. Traffic flows from the input stream through the Redact transform to tokenized output, and only the holder of the matching Decrypt configuration and Cloud KMS permissions can reverse the flow back to originals.

Key Facts

• Sensitive Data Protection (SDP) scans data where it is stored without moving data outside its region of origin, and an inspection template used for profiling must be in the same region as the profiled data.
• recordTransformations apply protection per field in structured tabular data, while infoTypeTransformations transform only detected findings in unstructured free text.
• Format-Preserving Encryption (FPE) preserves format and length, deterministic encryption preserves referential integrity by producing the same token for the same input, and cryptographic hashing is irreversible.
• SDP unwraps a Cloud KMS-wrapped data key at transformation time and performs encryption in memory, so the pipeline depends on both SDP and Cloud Key Management Service (Cloud KMS) authorization.
• K-anonymity means the quasi-identifiers for each person are identical to at least k minus 1 other people in the dataset, so each released record blends into a group of at least k records.