A data platform team is managing an enterprise data lake on Google Cloud where petabyte-scale datasets in BigQuery and Cloud Storage receive continuous updates. The team needs to implement an automated data quality and monitoring framework to meet the following requirements: Validate completeness and correctness across incoming daily data records without running costly full-table scans across historical partitions. Detect and profile any unmasked personally identifiable information (PII) introduced by source systems and publish findings directly to the metadata governance catalog. Continuously measure streaming pipeline processing delays against a service level objective (SLO) data freshness target and alert when latency spikes occur. Which architecture should the data platform team deploy to satisfy these requirements?

Configure Dataplex auto data quality scans with an incremental scope tracking the ingestion timestamp column, enable Sensitive Data Protection discovery to export data profiles to Dataplex Universal Catalog, and configure Cloud Monitoring alert policies on Dataflow data freshness metrics.: ### What the Solution Is This architecture combines **Dataplex auto data quality scans**, **Sensitive Data Protection (Cloud DLP) discovery**, and **Cloud Monitoring** with **Dataflow metrics** to establish an automated, scalable data lake monitoring framework. ### How It Meets the Requirements - **Incremental completeness and correctness validation**: Configuring Dataplex data quality scans with an **Incremental** scope uses a `DATE` or `TIMESTAMP` column to evaluate only newly appended data records. This avoids scanning entire multi-terabyte or petabyte tables, drastically reducing compute query costs and processing time. - **PII detection and catalog integration**: Enabling **Sensitive Data Protection discovery** continuously scans new and modified assets across BigQuery and Cloud Storage to detect sensitive infoTypes, classify risk levels, and automatically export rich data profiles into **Dataplex Universal Catalog**. - **Freshness monitoring and alerting**: Dataflow natively reports the `data_freshness` metric (measuring the age of the oldest unprocessed element) directly into **Cloud Monitoring**, enabling threshold-based alerting against defined freshness SLOs. ### Key Advantages - **Cost-optimized evaluation**: Incremental scans restrict validation queries to recent delta partitions. - **Unified metadata governance**: Publishes quality scan scores and sensitive data profiles natively to BigQuery and Dataplex Catalog. - **Automated compliance**: Continuously discovers undocumented PII across decentralized lake storage. - **Real-time observability**: Directly tracks pipeline latency Service Level Indicators (SLIs) in Cloud Monitoring. ### Why This Is Optimal This solution relies entirely on native, fully managed Google Cloud services designed for data governance, automated scanning, and streaming observability, avoiding custom script maintenance while ensuring minimal compute overhead.

Automated Data Quality Freshness Lake Monitoring | professional-cloud-data-engineer

Automated Data Quality Freshness Lake Monitoring

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A data platform team is managing an enterprise data lake on Google Cloud where petabyte-scale datasets in BigQuery and Cloud Storage receive continuous updates. The team needs to implement an automated data quality and monitoring framework to meet the following requirements:

Validate completeness and correctness across incoming daily data records without running costly full-table scans across historical partitions.
Detect and profile any unmasked personally identifiable information (PII) introduced by source systems and publish findings directly to the metadata governance catalog.
Continuously measure streaming pipeline processing delays against a service level objective (SLO) data freshness target and alert when latency spikes occur.

Which architecture should the data platform team deploy to satisfy these requirements?

Create Cloud Composer DAGs that execute Dataflow batch jobs with Join transforms to verify table consistency, configure Security Command Center built-in detectors for database encryption posture, and evaluate pipeline execution duration via Cloud Logging metrics.

Execute BigQuery scheduled queries with full table joins to validate row counts, trigger on-demand DLP inspection templates using Cloud Functions upon object finalization in Cloud Storage, and track Dataflow worker CPU utilization in Cloud Monitoring.

Deploy Dataproc ephemeral Spark jobs executing custom assertion scripts against full BigQuery tables, run synchronous DLP content.deidentify API calls on every batch output, and monitor pipeline health using Cloud Logging log sinks with BigQuery destinations.

Configure Dataplex auto data quality scans with an incremental scope tracking the ingestion timestamp column, enable Sensitive Data Protection discovery to export data profiles to Dataplex Universal Catalog, and configure Cloud Monitoring alert policies on Dataflow data freshness metrics.

Professional Cloud Data Engineer

Automated Data Quality Freshness Lake Monitoring

Related questions