An enterprise organization processes continuous streams of unstructured customer support transcripts using an automated data processing pipeline. Before feeding the text into a Vertex AI Foundation Model to generate sentiment analyses and topic summaries, the engineering team must satisfy the following technical and governance requirements: Sensitive Data Redaction: Detect and mask all personally identifiable information (PII) and payment data in-flight before the text reaches Vertex AI APIs. Automated Governance and Cataloging: Register the final enriched BigQuery datasets into an intelligent governance fabric without manually migrating or duplicating data assets. Lineage and Quality Tracking: Maintain end-to-end automated data lineage from ingestion to enrichment and enforce automated data quality monitoring on the enriched tables. Which pipeline and governance architecture should the team implement?

Use an Apache Beam pipeline in Dataflow with the Sensitive Data Protection (Cloud DLP) transform to de-identify text using a DeidentifyTemplate before calling Vertex AI, sink the enriched data to BigQuery, and govern the assets using Dataplex with automated data lineage and auto data quality tasks.: ### What This Architecture Is This architecture combines **Apache Beam on Dataflow**, the **Sensitive Data Protection (Cloud DLP) API**, **Vertex AI**, and **Dataplex Universal Catalog** to build a secure, real-time enrichment pipeline with end-to-end automated governance. ### How It Meets the Requirements - **Sensitive Data Redaction**: Dataflow natively incorporates Cloud DLP transforms that leverage a predefined `DeidentifyTemplate` to inspect, redact, or mask sensitive infoTypes (such as credit card numbers or national IDs) in-flight before passing sanitized payloads to the **Vertex AI** API. - **Automated Governance and Cataloging**: **Dataplex** discovers BigQuery tables automatically and unifies distributed assets into logical lakes and zones without requiring data movement or physical consolidation. - **Lineage and Quality Tracking**: Dataflow and BigQuery automatically publish metadata to **Dataplex Data Lineage**, while Dataplex **Auto Data Quality** tasks run automated validation rules against enriched BigQuery datasets. ### Key Advantages and Capabilities - **In-flight Sanitization**: Prevents sensitive customer data from being transmitted to third-party endpoints or foundation models. - **Serverless Autoscaling**: Dataflow handles dynamic scaling for streaming and batch enrichment workloads. - **Integrated Provenance**: Captures transformation steps from Pub/Sub to Dataflow and into BigQuery without custom lineage logging. - **Automated Quality Rules**: Executes scheduled and event-driven data validation checks directly on analytical storage. ### Why This Is Optimal This solution meets all privacy, enrichment, and governance requirements using fully managed Google Cloud native services, eliminating the need for custom microservices, brittle lineage scripts, or exposing raw PII to AI models.

Secure Govern Enriched Data Dlp Dataplex | professional-cloud-data-engineer

Secure Govern Enriched Data Dlp Dataplex

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise organization processes continuous streams of unstructured customer support transcripts using an automated data processing pipeline. Before feeding the text into a Vertex AI Foundation Model to generate sentiment analyses and topic summaries, the engineering team must satisfy the following technical and governance requirements:

Sensitive Data Redaction: Detect and mask all personally identifiable information (PII) and payment data in-flight before the text reaches Vertex AI APIs.
Automated Governance and Cataloging: Register the final enriched BigQuery datasets into an intelligent governance fabric without manually migrating or duplicating data assets.
Lineage and Quality Tracking: Maintain end-to-end automated data lineage from ingestion to enrichment and enforce automated data quality monitoring on the enriched tables.

Which pipeline and governance architecture should the team implement?

Feed raw transcripts directly from Pub/Sub into Vertex AI Batch Prediction, write raw predictions to BigQuery, execute scheduled DLP InspectContent jobs on BigQuery tables, and track data lineage by analyzing Cloud Logging audit logs.

Use an Apache Beam pipeline in Dataflow with the Sensitive Data Protection (Cloud DLP) transform to de-identify text using a DeidentifyTemplate before calling Vertex AI, sink the enriched data to BigQuery, and govern the assets using Dataplex with automated data lineage and auto data quality tasks.

Use BigQuery external tables to query raw transcripts in Cloud Storage, invoke Vertex AI via BigQuery ML remote functions, execute DLP ReidentifyContent upon querying, and organize datasets manually using BigQuery table labels.

Encrypt input text fields in Dataflow using Cloud KMS customer-managed encryption keys, send encrypted ciphertext to Vertex AI for inference, and monitor dataset health using Cloud Monitoring custom metric alerts.

Professional Cloud Data Engineer

Secure Govern Enriched Data Dlp Dataplex

Related questions