Protecting Sensitive Data with SDP Controls

Sensitive Data Protection (SDP) (opens in a new tab) is the Google Cloud service for discovering, classifying, and protecting sensitive data such as personally identifiable information (PII). Cloud Data Loss Prevention (Cloud DLP) is now part of Sensitive Data Protection, while the API name remains the Cloud Data Loss Prevention API (DLP API). SDP scans data where it is stored, so data does not leave its region of origin during inspection or profiling.

1.1Discovery Profiling, Inspection Jobs, and Inline Inspection

Discovery, inspection jobs, and synchronous inspection are three different ways SDP finds sensitive data. Discovery profiling continuously scans at the organization, folder, or project level and produces data profiles with sensitivity and data risk levels. Inspection jobs perform detailed on-demand scans of specific Cloud Storage buckets, BigQuery (opens in a new tab) tables, or Datastore kinds and return individual findings with quotes and exact locations. Synchronous content methods inspect data inline through the API before it is stored, which blocks or transforms sensitive data before it enters backend storage.

Discovery acts as the source of truth for where sensitive data lives, while an inspection job is used when investigation needs exact locations after discovery reports a high free-text score or multiple infoTypes. Job triggers automate inspection jobs on a schedule so repositories are rescanned as data changes. Discovery scan configurations can publish profile insights to Security Command Center, Google Security Operations, BigQuery profile copies, sample findings tables, and Pub/Sub for programmatic remediation.

1.2Inspection Templates, InfoTypes, and Detection Tuning

Inspection templates in SDP are reusable configurations that persist what to look for and how to look for it. A template stores selected infoTypes, optional rulesets, and a confidence threshold, and it can be referenced by inspection jobs and discovery scan configurations. A template is valid only when at least one infoType is selected, and the same template can be reused across Cloud Storage, BigQuery, and Firestore in Datastore mode. Templates are regional resources, and a template used for profiling must be in the same region as the data to be profiled.

InfoTypes are the classifiers that define what counts as sensitive data. Built-in infoTypes cover more than 100 categories such as PERSON_NAME, PHONE_NUMBER, US_SOCIAL_SECURITY_NUMBER, and DATE_OF_BIRTH, and they combine pattern matching with context signals. Custom infoTypes extend detection to organization-specific data with regular expressions, dictionaries and word lists, and stored infoTypes. Detection tuning controls the tradeoff between misses and false positives: raising the confidence threshold reduces noise but can miss weaker matches, hotword rules raise likelihood when a nearby word such as patient supports a PERSON_NAME match, and exclusion rules omit overlapping matches such as a DOMAIN_NAME inside an email address.

1.3Non-Cryptographic Redaction and Transformation Scopes

Non-cryptographic de-identification in SDP removes or generalizes sensitive values without encryption keys. Masking obscures characters in a detected value, redaction removes matched text or an image region and replaces it with a placeholder or opaque rectangle, and bucketing and date shifting generalize values to reduce identifiability. Because masking, redaction, bucketing, and date shifting are generally one-way, the sanitized replica cannot be reversed to recover originals. Choose these methods when sanitized data must flow to storage, export, or analytics and no authorized workflow needs the originals, and choose removal when high-risk data is not needed at all.

SDP divides de-identification work into two transformation scopes that control how inspection findings flow into protection. recordTransformations apply a transformation configuration per field and are designed for structured or tabular data, while infoTypeTransformations inspect for sensitive data and transform each finding and are designed for unstructured or free-text data. A per-field rule protects an entire column uniformly, such as transforming values in a US_SOCIAL_SECURITY_NUMBER column, while an infoType-based rule preserves surrounding narrative and changes only the detected span, such as masking an SSN inside a sentence. Both scopes can support custom infoTypes in addition to built-in detectors.

1.4Cryptographic Pseudonymization, Keys, and Re-Identification

Cryptographic pseudonymization in SDP replaces raw PII with tokens that preserve analytical utility while reducing exposure. Format-Preserving Encryption (FPE) (opens in a new tab) encrypts a value while keeping its original format and length, so a tokenized phone number still passes format checks. Deterministic encryption always produces the same token for the same input with the same key and configuration, which preserves referential integrity for joins and counts across rows and tables. Cryptographic hashing applies a one-way keyed function that cannot be decrypted, which removes re-identification risk but also removes recovery.

The transformation configuration uses a Cloud Key Management Service (Cloud KMS) wrapped key to protect the data-encryption key that actually tokenizes values. The kmsWrapped block contains a wrappedKey and a cryptoKeyName, and at transformation time SDP calls Cloud KMS to unwrap the key, performs encryption in memory, and writes only the token to output. Google recommends wrapped keys for sensitive workloads because unwrapped keys are suitable only for demonstration. The surrogate infoType labels the token, such as EMAIL_ADDRESS_TOKEN for EMAIL_ADDRESS, so SDP can find it later for re-identification. Only data encrypted with a reversible algorithm can be restored through projects.content.reidentify with the same wrapped key, surrogate infoType, and template, while hashed, masked, or redacted values cannot be re-identified.

1.5Text, Image, and Automated Pipeline De-Identification

Text classification, image redaction, and automated pipelines extend SDP protection to free-form and large-scale workloads. Classification of free-form input returns findings organized by InfoType, Likelihood, and Offset, where offset marks where the match was found, without changing the original text. Automatic text redaction produces sanitized output, for example replacing an email address and driver license with *** while sentence structure remains. The application flow is fixed: data flows from the input handler to SDP for inspection and redaction, then only the redacted text flows to the database, log, or model.

Image redaction detects sensitive text with object character recognition (OCR) and sensitive objects such as OBJECT_TYPE/BARCODE, OBJECT_TYPE/LICENSE_PLATE, and OBJECT_TYPE/WHITEBOARD, then returns a redacted copy obscured by opaque rectangles. A reference architecture separates a Dataflow (opens in a new tab) streaming de-identification pipeline, configuration management for templates and keys accessible only to security admins, and a validation and re-identification pipeline. Cloud Data Fusion (opens in a new tab) integrates the same controls through PII Filter, Redact, and Decrypt transforms, where PII Filter separates sensitive records, Redact applies masking or encryption, and Decrypt restores values encrypted with a reversible method. Statistical k-anonymity measures whether quasi-identifiers for each person are identical to at least k minus 1 others, so each released record blends into a group of at least k records.

Google Cloud data services combine identity-based access with perimeter and fine-grained controls to prevent unauthorized exposure. BigQuery (opens in a new tab) is the analytical warehouse with dataset, table, column, and row controls, Cloud Storage is the object store for many data types reachable from inside and outside Google Cloud, and Cloud SQL holds sensitive transactional data. Identity and Access Management (IAM) decides which principal has which role on which resource, while VPC Service Controls decides whether the request context allows it to cross a security perimeter.

2.1Enforce Access Boundaries and Object Protection for Cloud Storage

Object protection for Cloud Storage (opens in a new tab) uses two independent boundaries: IAM for identity-based access and VPC Service Controls for context-based perimeter defense. Uniform bucket-level access makes the IAM policy the single authority for buckets and objects, so a principal can upload or read data only with appropriate permissions checked on every access attempt. Centralizing authorization in IAM removes divergent per-object controls and lets tools such as Policy Analyzer verify who can read sensitive data.

VPC Service Controls (opens in a new tab) is a managed perimeter defense that can deny a request even when IAM allows it. A service perimeter forms a security boundary around resources such as Cloud Storage and BigQuery, and Access Context Manager access levels permit only designated principals based on attributes such as IP address and identity. Traffic flows from the requester to VPC Service Controls for boundary evaluation before it reaches the bucket or object, so copying data outside without a matching access level is blocked. Choose a single large perimeter around all sensitive projects for the most effective exfiltration protection with lowest overhead, and choose multiple perimeters only to isolate high-trust from de-identified data, regulated data, or independent tenants.

2.2Implement Fine-Grained Access Controls in BigQuery

BigQuery fine-grained access control restricts visibility below the dataset level to specific tables, columns, and rows. Organizations contain projects, projects contain datasets, and tables inherit permissions from the dataset that contains them. Table-level access grants a user, group, or service account access to a specific table or view without granting the entire dataset, using predefined roles for common patterns or custom roles for specific permissions.

Column-level security controls access to sensitive columns with policy tags organized in a taxonomy created in Data Catalog. A user must have dataset access plus the Data Catalog Fine-Grained Reader role on the relevant policy tag to read protected column data, while untagged columns remain available. A common tiered taxonomy defines 3_Confidential for credit card numbers, 2_Private for PII such as first name, and 1_Sensitive for non-public data such as credit limit, where higher tiers include lower tiers. Query behavior fails closed: a user without the tag grant who runs SELECT * receives an error naming the required tag and can retry with SELECT * EXCEPT (ssn). Policy tags apply only to leaf fields of STRUCT or RECORD columns, require GoogleSQL, and to clear a tag the schema policyTags.names must be set to [].

Row-level security filters table data so a principal sees only rows that satisfy a qualifying user condition evaluated at query time. An authorized view is authorized to read its source dataset, so users can query view results without access to underlying source data while base-table column policies still apply. There is a hard limit of 2500 authorized resources per dataset, including authorized views, datasets, and functions.

2.3Secure Cloud SQL Connectivity and Database Authentication

Cloud SQL (opens in a new tab) secure connectivity removes public network paths, requires encrypted transit, and allows only authorized identities to perform work. Private IP connectivity means clients reach the instance through a Virtual Private Cloud (VPC) network instead of the public internet, so traffic never leaves private boundaries. Public access is restricted by preventing public IP assignment, auditing existing public IPs, and allowlisting hybrid addresses when needed, with Cloud VPN or Cloud Interconnect protecting hybrid paths. Firewall rules deny all traffic by default and allow only defined ports and sources, while VPC Service Controls defines a perimeter so only API traffic from inside can reach restricted services.

Enforced SSL/TLS encryption means every incoming database connection must use cryptographic protection for confidentiality and integrity. The minimum standard is TLS 1.2 or higher, with legacy versions restricted to prevent downgrade, and remote sessions use SSH or HTTPS rather than cleartext. For sensitive instances, transport encryption is paired with customer-managed encryption keys (CMEKs) in Cloud KMS so data at rest is encrypted under customer-controlled keys. Cloud IAM database authentication grants administrative and transactional access only through least-privilege roles, with separation of duties, principal access boundary policies, Privileged Access Manager for just-in-time elevation, and restricted service-account key creation.

Google Secret Manager (opens in a new tab) lifecycle management controls a secret from creation through version replacement, rotation, and expiration. A secret is the named container such as projects/PROJECT_ID/locations/LOCATION/secrets/SECRET_ID, and each secret version is an immutable copy of the credential value inside it. Continuous hygiene depends on managing version states, running rotation on a schedule, and enforcing expiration so stale credentials cannot live forever.

3.1Secret Versions and Lifecycle States

Secret versions in Secret Manager are single immutable values with their own resource name, creation time, and state. Versions appear in the ENABLED state after creation or rotation and in the DISABLED state after a disable operation. A disabled version cannot be used by future access requests until re-enabled, while other versions of the same secret are unaffected. This state model allows safe rotation because the workflow depends on the new version becoming ENABLED while the prior version is retained in DISABLED form for rollback.

Secret values can be in any format but must not be larger than 64 KiB, and the first value uploaded at creation becomes the first version. For specialized use, a regional secret can be created with a fixed secretType such as CLOUD_SQL_DB_CREDENTIALS, which tells Secret Manager that rotation must also update the linked Cloud SQL instance. The secret type cannot be added or changed on an existing secret, so the type decision must be made at creation.

3.2Rotation Schedules and Expiration Policies

A rotation schedule in Secret Manager is a start time plus a recurring period that defines when the value must be replaced. In the console the schedule uses a Starting on date and time, after which the timer triggers replacement at each interval. For regional secrets the manual trigger is the rotateSecret method on secretmanager.LOCATION.rep.googleapis.com. When rotation runs for a Cloud SQL credential secret, Secret Manager first creates a new version and then attempts to update credentials in the Cloud SQL instance, and only on success does Rotation status change to Enabled. An event-driven design extends this flow where a Pub/Sub (opens in a new tab) notification carries the rotation event to a Cloud Functions handler that generates the new credential and propagates it before the old version is disabled.

An expiration policy is a time limit after which the entire secret is no longer usable. Rotation maintains freshness during the lifetime, while expiration guarantees termination at the end of it.

The gcloud secrets create commands reflect this choice through --expire-time "TIMESTAMP" versus --ttl "DURATION". Expiration (opens in a new tab) applies to the secret container while rotation creates new versions inside it, so applications that depend on the secret must refresh it before expiration to avoid outages.

3.3Granular IAM and Authentication Scopes for Secret Access

Granular IAM for Secret Manager means granting access per secret to a specific service account rather than broad project access. A binding attaches roles/secretmanager.secretAccessor on a single secret resource to one member, so only that workload identity can read that version. Best practices include blocking automatic grants to default service accounts, creating Google Kubernetes Engine (GKE) clusters with limited privileges, and removing or replacing per-secret bindings when workloads change. Secret details (opens in a new tab) and bindings are managed per secret so no other secret or workload is affected.

Authentication scopes are a second gate for Compute Engine and GKE workloads. The instance or node pool must be provisioned with the cloud-platform OAuth scope, https://www.googleapis.com/auth/cloud-platform, or calls fail with Request had insufficient authentication scopes even when IAM is correct. Traffic flows from application code through the client library to the Secret Manager API only after both checks pass: the service account holds IAM permission and the host holds the scope. Enabling access follows an ordered dependency: first enable the Secret Manager API, then grant secret-level IAM to the workload service account, then provision the host scope at creation or with gcloud compute instances set-service-account.

3.4Encryption, Replication, and Audit Logging

Customer-managed encryption keys (CMEK) in Secret Manager replace Google default encryption when compliance requires customer control. With Google default encryption, Google owns and manages the key encryption keys under envelope encryption with no customer action. With CMEK, the customer creates a key ring and symmetric key in Cloud KMS and controls location, rotation, access, boundaries, and audit. Access flows from Secret Manager through a Google-managed service agent to Cloud KMS, and the service agent must hold roles/cloudkms.cryptoKeyEncrypterDecrypter on the key. The key location must match the secret location, CMEK must be selected at creation, and organization policy constraints/gcp.restrictNonCmekServices can refuse unprotected resources.

A replication policy defines where secret replicas reside and determines residency compliance. Automatic replication lets Google replicate within its footprint, while user-managed replication pins replicas to customer-selected regions. Replication and CMEK location depend on each other because the key ring, key, and replicas must share the same effective location. Authentication (opens in a new tab) and audit logging answer who accessed which secret: Cloud Audit Logs record administration and access, while Cloud KMS Data Access logs record Encrypt and Decrypt calls. Secret Manager polls the key periodically, so a healthy key produces about two entries every five minutes per resource, and a break, surge, or FAILED_PRECONDITION indicates revoked access or anomalous use.

3.5Network Perimeters and Private Connectivity

VPC Service Controls is the network-layer boundary that prevents exfiltration of Secret Manager API calls. A service perimeter encloses selected projects and defines allowed services, so a request from an authorized service account inside the perimeter to Secret Manager inside the same perimeter is permitted, while a copy directed outside is denied. Access Context Manager policies add context such as network origin and identity before allowing cross-boundary access. Private Google Access for VPC subnets lets instances without external IPs reach Google APIs including Secret Manager without traversing the public internet. Complementary guardrails include blocking external IP assignment, restricting default network creation, enabling private GKE clusters, enforcing deny-all egress with narrow allows, and restricting inbound to SSH port 22 and RDP port 3389 unless explicitly required.

Metadata security controls are the IAM, service account, shell-access, and guest-firewall settings that limit who can manage virtual machine (VM) metadata and which processes can read it. Every Compute Engine (opens in a new tab) instance can query the metadata server automatically without extra authorization, and the request and response never leave the physical host that runs the VM. Any process that can reach the metadata query URL can read all values, including custom metadata, certificates, and keys, so both outside management and inside guest access must be restricted.

4.1Metadata Trust Boundary and Sensitive Data Exposure

The metadata trust boundary is the physical host that runs the VM. Traffic between the VM and the metadata server never leaves that host, but inside the guest there is no per-user check by default. A workload can request service account tokens from the metadata server, and any local process with network access to the query URL receives the same full set of values. Google recommends caution when writing sensitive values to metadata, sandboxing processes that should not have access, and avoiding confidential hints in service account email addresses.

For example, jenkins@deployment-project-123.iam.gserviceaccount.com reveals both project ID and software type, while deployer@deployment-project-123.iam.gserviceaccount.com avoids disclosing the software. Viewing an allow policy in another project can expose those emails because many roles including basic Viewer include *.getIamPolicy. If a service account is granted access to a less-controlled project, its email becomes visible more widely. The destination project controls the granting policy, not the owning project, so generic names matter for cross-project access.

4.2Restricting Metadata Management With IAM and Service Accounts

IAM (opens in a new tab) controls metadata management from outside the VM at project, zone, instance, and service-account levels. Querying metadata from inside requires only permission to connect to the instance, while viewing or changing metadata from console, CLI, or REST API requires Compute Engine and service-account permissions. When VMs use service accounts, callers also need iam.serviceAccounts.actAs, enforced through Service Account User and Compute Instance Admin roles.

Do not manage service account access at project or folder level when least privilege is required. Granting Service Account Token Creator at project level lets a user impersonate any service account in that project and potentially reach outside resources. Manage access on each individual service account instead of inheriting broad access from folders or the organization node.

4.3Hardening Shell Access Tied to the Attached Service Account

Shell access to a VM with a privileged service account is equivalent to the ability to act as that service account. A shell user can query the metadata server for access and ID tokens, so untrusted code creates the same risk. Risky sources include application code, user-submitted server-side scripts, code pulled from remote repositories, startup scripts from Cloud Storage, and guest policies from VM Manager. Remote code sources must be at least as well secured as the attached service account.

OS Login (opens in a new tab) enforces a stronger binding on Linux: to connect, the user must be allowed to use OS Login and must also hold iam.serviceAccounts.actAs on the attached service account. Identity-Aware Proxy (IAP) provides an additional gate for VMs without OS Login by requiring the IAP-Secured Tunnel User role only for users who should authenticate as the attached account. Metadata-based SSH keys and Windows passwords check metadata access plus iam.serviceAccounts.actAs only when the key is published or credential requested, with no IAM check on later logins.

4.4Limiting In-Guest Access and Handling Operational Limits

In-guest access to the metadata server is unrestricted by default. Even low-privilege identities such as nobody on Linux or LocalService on Windows can obtain tokens for the attached service account. The fix is enforced in the guest firewall, which filters outbound connections to the metadata server by user or group before the request leaves the VM. On Linux an iptables rule with --uid-owner and --gid-owner applies only to selected users or groups, while on Windows a firewall rule customized with Set-NetFirewallSecurityFilter does the same. Combine filtering with process sandboxing when third-party code runs on the same VM as a privileged account.

Metadata endpoints expose the same entries over HTTP at http://metadata.google.internal/computeMetadata/v1 and http://169.254.169.254/computeMetadata/v1, with Shielded VMs also supporting HTTPS in Preview. Host maintenance can interrupt the server for less than one second with HTTP Error 503, so applications should implement retry logic. Calls from inside do not incur egress charges or consume compute.googleapis.com quotas, though internal throttling still applies. Storage is bounded: 512 KB combined total for all entries, 128 bytes per key, 256 KB per value, with SSH keys and inline startup scripts counting toward those limits.

4.5Evaluate Attack Vectors and Enforce Secure Access

Server-Side Request Forgery (SSRF) is the primary way an external attacker reaches the internal-only metadata service. In an SSRF attack, traffic flows from the attacker to a vulnerable web application on the VM, then from that application to the metadata service, then back to the attacker in the HTTP response. The metadata service trusts the request because it comes from inside the VM boundary, even though the attacker controls it from outside. The attacker needs only the application to fetch an attacker-supplied URL, not inbound SSH access.

Misconfigured firewall rules create external exposure by allowing inbound traffic from 0.0.0.0/0 to reach the VM. Findings (opens in a new tab) include open SSH, RDP, Telnet, SMTP, database, and NetBIOS ports, and observed exploitation has led to credential harvesting and cryptomining. Credential theft then progresses through impersonation, privilege expansion, persistence via SSH keys or startup scripts, and exfiltration. Security Health Analytics observes posture, Web Security Scanner observes web flaws such as SSRF, and Event Threat Detection observes log-based behavior such as anomalous delegation and Cloud SQL exfiltration. Least privilege breaks the chain by ensuring a stolen token grants only smallest required roles.

Secure access means only on-instance code can reach 169.254.169.254 while the internet cannot reach the instance. The remediation order is explicit: first configure Private-IP-only addressing or restrict firewall ingress, then create a Google Cloud Armor policy to block exploitation, then enable logging and threat detection. Access control (opens in a new tab) depends on custom VPC networks, deny-all egress with narrow allows, and blocking external IP assignment. Choose Private-IP-only when workloads can operate behind NAT or private paths, and reserve firewall-restricted public access for documented ingress needs because a rule error can re-expose ports.

Key Facts

• Sensitive Data Protection scans data where it is stored without leaving its region, and an inspection template used for profiling must be in the same region as the profiled data.
• recordTransformations protect per field in structured tabular data while infoTypeTransformations transform only detected findings in unstructured free text.
• Format-Preserving Encryption preserves format and length, deterministic encryption preserves referential integrity with the same token for the same input, and cryptographic hashing is irreversible.
• BigQuery column-level security requires dataset access plus the Data Catalog Fine-Grained Reader role on the policy tag, and authorized views are limited to 2500 authorized resources per dataset.
• Secret versions are immutable, limited to 64 KiB, and move between ENABLED and DISABLED states, while Compute Engine limits all metadata entries to 512 KB combined.