Advanced
Prepare and test your skills
Advanced
Prepare and test your skills
In-Flight PII Redaction for AI Enrichment Pipelines
Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise organization processes continuous streams of unstructured customer support transcripts using an automated data processing pipeline. Before feeding the text into a Vertex AI Foundation Model to generate sentiment analyses and topic summaries, the engineering team must satisfy the following technical and governance requirements:
- Sensitive Data Redaction: Detect and mask all personally identifiable information (PII) and payment data in-flight before the text reaches Vertex AI APIs.
- Automated Governance and Cataloging: Register the final enriched BigQuery datasets into an intelligent governance fabric without manually migrating or duplicating data assets.
- Lineage and Quality Tracking: Maintain end-to-end automated data lineage from ingestion to enrichment and enforce automated data quality monitoring on the enriched tables.
Which pipeline and governance architecture should the team implement?
Explanations
Use an Apache Beam pipeline in Dataflow with the Sensitive Data Protection (Cloud DLP) transform to de-identify text using a DeidentifyTemplate before calling Vertex AI, sink the enriched data to BigQuery, and govern the assets using Dataplex with automated data lineage and auto data quality tasks.
Use an Apache Beam pipeline in Dataflow with the Sensitive Data Protection (Cloud DLP) transform to de-identify text using a DeidentifyTemplate before calling Vertex AI, sink the enriched data to BigQuery, and govern the assets using Dataplex with automated data lineage and auto data quality tasks.
What This Architecture Is
This architecture combines Apache Beam on Dataflow, the Sensitive Data Protection (Cloud DLP) API, Vertex AI, and Dataplex Universal Catalog to build a secure, real-time enrichment pipeline with end-to-end automated governance.
How It Meets the Requirements
- Sensitive Data Redaction: Dataflow natively incorporates Cloud DLP transforms that leverage a predefined
DeidentifyTemplateto inspect, redact, or mask sensitive infoTypes (such as credit card numbers or national IDs) in-flight before passing sanitized payloads to the Vertex AI API. - Automated Governance and Cataloging: Dataplex discovers BigQuery tables automatically and unifies distributed assets into logical lakes and zones without requiring data movement or physical consolidation.
- Lineage and Quality Tracking: Dataflow and BigQuery automatically publish metadata to Dataplex Data Lineage, while Dataplex Auto Data Quality tasks run automated validation rules against enriched BigQuery datasets.
Key Advantages and Capabilities
- In-flight Sanitization: Prevents sensitive customer data from being transmitted to third-party endpoints or foundation models.
- Serverless Autoscaling: Dataflow handles dynamic scaling for streaming and batch enrichment workloads.
- Integrated Provenance: Captures transformation steps from Pub/Sub to Dataflow and into BigQuery without custom lineage logging.
- Automated Quality Rules: Executes scheduled and event-driven data validation checks directly on analytical storage.
Why This Is Optimal
This solution meets all privacy, enrichment, and governance requirements using fully managed Google Cloud native services, eliminating the need for custom microservices, brittle lineage scripts, or exposing raw PII to AI models.
Feed raw transcripts directly from Pub/Sub into Vertex AI Batch Prediction, write raw predictions to BigQuery, execute scheduled DLP InspectContent jobs on BigQuery tables, and track data lineage by analyzing Cloud Logging audit logs.
Feed raw transcripts directly from Pub/Sub into Vertex AI Batch Prediction, write raw predictions to BigQuery, execute scheduled DLP InspectContent jobs on BigQuery tables, and track data lineage by analyzing Cloud Logging audit logs.
What This Option Is
This approach sends unredacted data directly to Vertex AI for batch inference and relies on post-hoc Cloud DLP inspection jobs on BigQuery, combined with manual log parsing in Cloud Logging for tracking transformations.
Why It Doesn't Meet the Requirement
- Fails Sensitive Data Redaction: Transmitting raw, unmasked transcripts directly to Vertex AI violates the requirement to mask PII prior to AI processing.
- Fails Automated Governance and Lineage: Cloud Logging records API audit logs but does not construct an automated, visual data lineage graph or manage business metadata.
- Fails Quality Tracking: Inspecting data after inference exposes downstream analytical consumers to unredacted data and does not enforce automated data quality validation rules.
Key Limitations
- Raw PII reaches the machine learning model layer unshielded.
- Audit logs in Cloud Logging lack native graph-based lineage visualization.
- Scheduled post-processing DLP scans create a window of exposure where sensitive data sits unprotected.
This Answer Would Be Correct If
This design would be suitable only for offline, non-sensitive batch analytics where real-time de-identification is not required and external model compliance boundaries do not apply.
Encrypt input text fields in Dataflow using Cloud KMS customer-managed encryption keys, send encrypted ciphertext to Vertex AI for inference, and monitor dataset health using Cloud Monitoring custom metric alerts.
Encrypt input text fields in Dataflow using Cloud KMS customer-managed encryption keys, send encrypted ciphertext to Vertex AI for inference, and monitor dataset health using Cloud Monitoring custom metric alerts.
What This Option Is
This approach uses Cloud Key Management Service (Cloud KMS) to perform field-level encryption on text payloads within Dataflow before calling Vertex AI, while relying on Cloud Monitoring for pipeline observability.
Why It Doesn't Meet the Requirement
- Fails Sensitive Data Redaction: Encrypting text into ciphertext renders the input unreadable to Vertex AI Natural Language and Foundation Models, causing model inference to fail or produce garbage output.
- Fails Automated Governance and Cataloging: Cloud Monitoring tracks performance metrics (such as latency and throughput) but does not provide cataloging, data search, or semantic metadata governance.
- Fails Lineage and Quality Tracking: Cloud Monitoring cannot generate data lineage graphs or validate table-level data quality rules.
Key Limitations
- Foundation models require intelligible natural language text, not cryptographic ciphertexts.
- Cloud KMS provides cryptographic access control, not intelligent entity detection or de-identification.
- Monitoring metrics cannot substitute for data cataloging and schema validation.
This Answer Would Be Correct If
This pattern would be appropriate for storing sensitive data at rest where authorized downstream microservices possess the KMS key to decrypt fields before consumption.
Use BigQuery external tables to query raw transcripts in Cloud Storage, invoke Vertex AI via BigQuery ML remote functions, execute DLP ReidentifyContent upon querying, and organize datasets manually using BigQuery table labels.
Use BigQuery external tables to query raw transcripts in Cloud Storage, invoke Vertex AI via BigQuery ML remote functions, execute DLP ReidentifyContent upon querying, and organize datasets manually using BigQuery table labels.
What This Option Is
This design loads raw transcripts via BigQuery external tables, executes BigQuery ML remote functions to call Vertex AI, attempts to apply DLP re-identification on queries, and relies on table labels for cataloging.
Why It Doesn't Meet the Requirement
- Fails In-Flight Redaction: Raw transcripts stored in Cloud Storage are sent directly to the BigQuery ML remote function and Vertex AI without pre-inference masking.
- Fails Automated Governance: Static BigQuery table labels do not constitute an active governance fabric and require continuous manual maintenance.
- Fails Automated Lineage and Quality: This setup lacks native data quality validation pipelines and does not provide comprehensive cross-system lineage tracking out of the box.
Key Limitations
- External tables on raw text files exhibit slower performance compared to native streaming pipelines.
ReidentifyContentis used to restore masked data to plaintext, not to protect data prior to AI processing.- Table labels do not support automated policy enforcement or discovery across multi-cloud or multi-project environments.
This Answer Would Be Correct If
This pattern would be appropriate for lightweight, ad-hoc analytical queries on non-sensitive, already-curated data warehouses.
Try 10 interactive questions like this one, free
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
- 1,353 practice questions across all 5 exam domains
- Every answer explained: why it is right, and why the others are not
- Scout, an AI tutor that answers your follow-up questions
- Readiness tracking by domain, so you know when to book the exam
Related questions
Keep the momentum going with these hand-picked practice scenarios
AdvancedProfessional Data Engineer
BigQuery ML Remote Models with Vertex AI
AI data enrichment
AdvancedProfessional Data Engineer
Dataflow GroupIntoBatches API Quota Management
AI data enrichment
AdvancedProfessional Data Engineer
Cloud Spanner INSERT_OR_UPDATE Idempotent Writes
Streaming (e.g., windowing, late arriving data)
AdvancedProfessional Data Engineer
BigQuery ELT vs Dataproc ETL
Defining data transformation and orchestration logic
AdvancedProfessional Data Engineer
Pub/Sub Dead-Letter Topics and Snapshots
Making decisions related to ACID (atomicity, consistency, isolation, and durability) compliance and availability
FundamentalAssociate Data Practitioner
BigQuery %%bigquery Cell Magic DataFrame
Analyze data to answer business questions
Last updated Validated