Discover and Inspect Sensitive Data Across Cloud Workloads

Data Discovery and Profiling

Sensitive Data Protection (SDP) offers two key services. The first is discovery, a continuous scanning service that builds data profiles of your information across Cloud Storage, BigQuery, and Datastore. These profiles give you a high-level view of where sensitive data resides and its risk level across your entire organization, folder, or project. You can automate this by setting up organization-level discovery profiles, which automatically scan and classify new data as it is created, providing a central source of truth for security audits.

Data Inspection and Tuning

To find specific pieces of sensitive data, you run an inspection job on a single resource, like a table or bucket. This job uses an inspection template, which defines what to look for using built-in infoTypes (like US_SOCIAL_SECURITY_NUMBER) or custom infoTypes you create using regular expressions or dictionaries. After a scan, you need to evaluate the findings. To make the inspection more accurate, you can tune it by adjusting the likelihood threshold to filter out low-confidence results and creating hotword rules to only flag data when specific context is present (like the word "SSN" next to a number). For data sources not directly supported, you can export the data to a supported format like CSV in Cloud Storage for inspection.

Implement Data Redaction, Masking, and Statistical De-Identification

Non-Cryptographic Transformations

When sensitive data is found, SDP can apply various techniques to sanitize it without using encryption, which helps preserve the data's usefulness for analysis. Character masking replaces sensitive text with placeholder characters (like ****). Date shifting randomly changes dates within a set range, keeping the order of events intact for time-based analysis. Record suppression can remove entire rows of data from a dataset if they meet certain risky conditions. These methods allow data to be shared or analyzed more safely.

Redacting Unstructured Data and Automating Pipelines

SDP can also protect sensitive information in images using optical character recognition (OCR). The image.redact API can find text, barcodes, or license plates in an image and cover them with opaque boxes before the image is saved or shared. To apply these protections at scale, you integrate SDP into automated data pipelines. For example, you can use native plugins in Cloud Data Fusion to filter or mask data as it flows, or use BigQuery remote functions to run SDP transformations directly within your SQL queries. This ensures data is automatically cleaned before it is stored, exported, or analyzed.

Configure Pseudonymization and Cryptographic Transformations

Pseudonymization and Format-Preserving Encryption

Pseudonymization replaces sensitive data with artificial tokens, but allows you to reverse the process later if needed. This maintains referential integrity, meaning you can still link related data across different systems without exposing the original information. A key technique is Format-Preserving Encryption (FPE), which encrypts data but keeps it in the same format. For instance, an encrypted credit card number still looks like a valid card number, which helps it work with existing databases and applications without requiring changes.

Key Management and Reversibility

These cryptographic transformations rely on keys managed by Cloud Key Management Service (Cloud KMS). The process uses envelope encryption: an encryption key is itself encrypted (or "wrapped") by a Cloud KMS key. This wrapped key is then used by SDP to transform your data, keeping the core encryption keys secure. Reversible transformations like FPE allow for re-identification—getting the original data back—when you have the correct key. During de-identification, you assign a surrogate info type as a marker, so the system knows which data can be reversed later.

Choosing the Right Transformation

Deciding between reversible methods (like FPE) and irreversible methods (like one-way hashing) involves a trade-off. Reversible pseudonymization preserves more analytical utility, letting you join datasets and perform complex analysis, but it requires strict key management. Irreversible methods offer stronger security since the original data cannot be recovered, but they limit how the data can be reused. Your choice depends on your specific need for data analysis, your compliance requirements, and your overall security goals.