An enterprise data platform ingests streaming customer interaction records and log files continuously into Cloud Storage, while also running batch analytical transformations in BigQuery. The security and compliance team has established the following requirements for sensitive data governance: Scan incoming and existing data for standard personally identifiable information (PII) including email addresses and phone numbers. Detect internal customer account identifiers that adhere to a specific pattern (for example, ACC-[0-9]{8}). Detect sensitive keywords against a managed master list containing over 500,000 proprietary customer account aliases stored in Cloud Storage. Maintain centralized, reusable inspection definitions across all continuous ingestion pipelines, storage scanners, and BigQuery batch jobs. Store detailed finding metadata, including confidence levels and location coordinates, for automated compliance auditing. Which architecture and configuration should the data engineering team implement using Sensitive Data Protection (Cloud DLP)?

Create a regex custom infoType for account identifiers and a StoredInfoType backed by a large custom dictionary for the alias list. Combine these custom detectors with built-in PII infoTypes inside a centralized InspectTemplate, reference the template in your pipelines and job triggers, and configure actions to publish findings to BigQuery.: What This Architecture Is This architecture leverages Sensitive Data Protection (Cloud DLP) core building blocks—Custom InfoTypes, StoredInfoTypes, Inspection Templates (InspectTemplate), and Finding Actions—to deliver scalable, centralized, and automated sensitive data discovery across diverse storage systems and streaming ingestion pipelines. How It Meets The Requirements Standard PII Detection: Built-in infoType detectors (such as EMAIL_ADDRESS and PHONE_NUMBER) are native to the inspection engine and require no custom regex maintenance. Custom Identifier Patterns: Defining a CustomInfoType with a regular expression (regex) rule cleanly captures standardized organizational patterns such as ACC-[0-9]{8}. Large Master Keyword List: Managing over 500,000 terms exceeds inline payload limits. A StoredInfoType configured with a LargeCustomDictionaryConfig referencing a file set in Cloud Storage indexes large term dictionaries efficiently for high-throughput matching. Centralized and Consistent Rules: An InspectTemplate decouples scanning rules from pipeline code. Any update made to the inspection template instantly propagates across Cloud Storage ingestion triggers, Dataflow pipelines, and BigQuery scheduled scans. Audit Findings Export: Configuring the inspection action to Save to BigQuery records granular finding metadata (confidence/likelihood, location offsets, column names, and infoType classifications) into persistent BigQuery tables for analysis. Key Advantages Decoupled Governance: Security administrators manage classification rules centrally in templates without requiring code redeployments in streaming pipelines. Scalable Dictionary Matching: StoredInfoType pre-indexes massive term lists, reducing inspection latency. Unified Multi-Cloud / Hybrid Coverage: Works identically across Cloud Storage objects, BigQuery tables, and relational data streams. Auditing and Traceability: Detailed BigQuery output allows security teams to run SQL queries and build Looker dashboards for compliance posture tracking. Why This Is Optimal Using centralized inspection templates with stored custom infoTypes avoids hardcoded rules in pipeline code, accommodates large-scale dictionary sizes, and natively outputs detailed audit trails directly to BigQuery without custom integration overhead.

Configure Dlp Inspection Templates Custom Infotypes | Professional Cloud Data Engineer

StudyCerts

Explanations

What This Architecture Is

This architecture leverages Sensitive Data Protection (Cloud DLP) core building blocks—Custom InfoTypes, StoredInfoTypes, Inspection Templates (InspectTemplate), and Finding Actions—to deliver scalable, centralized, and automated sensitive data discovery across diverse storage systems and streaming ingestion pipelines.

How It Meets The Requirements

Standard PII Detection: Built-in infoType detectors (such as EMAIL_ADDRESS and PHONE_NUMBER) are native to the inspection engine and require no custom regex maintenance.
Custom Identifier Patterns: Defining a CustomInfoType with a regular expression (regex) rule cleanly captures standardized organizational patterns such as ACC-[0-9]{8}.
Large Master Keyword List: Managing over 500,000 terms exceeds inline payload limits. A StoredInfoType configured with a LargeCustomDictionaryConfig referencing a file set in Cloud Storage indexes large term dictionaries efficiently for high-throughput matching.
Centralized and Consistent Rules: An InspectTemplate decouples scanning rules from pipeline code. Any update made to the inspection template instantly propagates across Cloud Storage ingestion triggers, Dataflow pipelines, and BigQuery scheduled scans.
Audit Findings Export: Configuring the inspection action to Save to BigQuery records granular finding metadata (confidence/likelihood, location offsets, column names, and infoType classifications) into persistent BigQuery tables for analysis.

Key Advantages

Decoupled Governance: Security administrators manage classification rules centrally in templates without requiring code redeployments in streaming pipelines.
Scalable Dictionary Matching: StoredInfoType pre-indexes massive term lists, reducing inspection latency.
Unified Multi-Cloud / Hybrid Coverage: Works identically across Cloud Storage objects, BigQuery tables, and relational data streams.
Auditing and Traceability: Detailed BigQuery output allows security teams to run SQL queries and build Looker dashboards for compliance posture tracking.

Why This Is Optimal

Using centralized inspection templates with stored custom infoTypes avoids hardcoded rules in pipeline code, accommodates large-scale dictionary sizes, and natively outputs detailed audit trails directly to BigQuery without custom integration overhead.

View source documentation

Professional Cloud Data Engineer

Configure Dlp Inspection Templates Custom Infotypes