A data engineering team is implementing a data cleansing and quality governance framework for raw transaction data ingested daily into BigQuery. To prevent data corruption from impacting downstream analytics, the team must meet the following requirements: Run visual data profiling on datasets to detect column distributions, null counts, anomalies, and outliers. Define and automate continuous data quality validation rules (including numeric range checks, regex pattern matches, and custom SQL assertions). Export detailed execution metrics, rule evaluation statuses, and queries for failed records to BigQuery for auditing. Automatically trigger notifications if data quality scores fall below a defined threshold. Which solution should the team implement in Dataplex to satisfy these requirements?

Configure Dataplex DataScan jobs to execute Data Profile scans for distribution insights, along with Data Quality scans utilizing dataQualitySpec rules, BigQuery post-scan export, and notification score thresholds.: What Dataplex DataScan is Dataplex DataScan is a fully managed, serverless governance capability in Google Cloud that automates data profiling and data quality validation directly across BigQuery tables and Cloud Storage data lakes without requiring data movement. How it meets the requirements Detect anomalies and outliers: Dataplex Data Profile scans (dataProfileSpec) compute column-level summary statistics, distinct values, null percentages, and distributions, allowing engineers to visually assess anomalies and data distributions directly within the Google Cloud console. Automated continuous data quality validation: Dataplex Data Quality scans (dataQualitySpec) execute declarative rules, such as rangeExpectation, regexExpectation, setExpectation, and custom sqlAssertion checks against datasets on a defined schedule. Export metrics and failed record queries: Post-scan actions (postScanActions.bigqueryExport.resultsTable) write comprehensive execution metadata—including rows_validated, success_percentage, failed_count, and a diagnostic failed_records_query SQL statement—directly into a target BigQuery table for historical auditing. Threshold-based notifications: Configuring postScanActions.notificationReport.scoreThresholdTrigger triggers automated email alerts when the overall quality score drops below the specified tolerance. Key advantages Serverless execution: Runs directly using BigQuery compute resources without managing separate VM clusters or ETL runtimes. Built-in diagnostics: Automatically generates SQL queries to extract the exact failed records for rapid troubleshooting. Centralized governance: Integrates natively with the Dataplex Universal Catalog to publish data quality aspects alongside technical metadata. Standardized metrics: Categorizes results across standardized data quality dimensions such as Completeness, Validity, and Uniqueness. Why this is optimal This approach provides an end-to-end, native Google Cloud framework for profiling, automated quality rule enforcement, auditing, and alert management, minimizing operational overhead while establishing automated data reliability.

StudyCerts

Professional Cloud Data Engineer

professional-cloud-data-engineer

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Dataplex Data Profiling Quality Scans Monitoring

Section: Data cleansing

A data engineering team is implementing a data cleansing and quality governance framework for raw transaction data ingested daily into BigQuery. To prevent data corruption from impacting downstream analytics, the team must meet the following requirements:

Run visual data profiling on datasets to detect column distributions, null counts, anomalies, and outliers.
Define and automate continuous data quality validation rules (including numeric range checks, regex pattern matches, and custom SQL assertions).
Export detailed execution metrics, rule evaluation statuses, and queries for failed records to BigQuery for auditing.
Automatically trigger notifications if data quality scores fall below a defined threshold.

Which solution should the team implement in Dataplex to satisfy these requirements?

Configure Dataplex DataScan jobs to execute Data Profile scans for distribution insights, along with Data Quality scans utilizing dataQualitySpec rules, BigQuery post-scan export, and notification score thresholds.

Execute BigQuery ML data validation functions (ML.DESCRIBE_DATA and ML.VALIDATE_DATA_DRIFT) via scheduled queries, logging error counts into custom audit tables.

Use Sensitive Data Protection (DLP) discovery scans to profile dataset columns, and configure Security Command Center finding posture policies to enforce data quality ranges.

Deploy Looker Studio connected to BigQuery to visualize data nulls and write custom SQL conditional alerts that send webhook notifications to pipeline orchestrators.

Professional Cloud Data Engineer

professional-cloud-data-engineer

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Advanced

Prepare and test your skills

Track Readiness

Set Exam Goal

Dataplex Data Profiling Quality Scans Monitoring

Section: Data cleansing

Run visual data profiling on datasets to detect column distributions, null counts, anomalies, and outliers.
Define and automate continuous data quality validation rules (including numeric range checks, regex pattern matches, and custom SQL assertions).
Export detailed execution metrics, rule evaluation statuses, and queries for failed records to BigQuery for auditing.
Automatically trigger notifications if data quality scores fall below a defined threshold.

Which solution should the team implement in Dataplex to satisfy these requirements?

Execute BigQuery ML data validation functions (ML.DESCRIBE_DATA and ML.VALIDATE_DATA_DRIFT) via scheduled queries, logging error counts into custom audit tables.

Use Sensitive Data Protection (DLP) discovery scans to profile dataset columns, and configure Security Command Center finding posture policies to enforce data quality ranges.

Deploy Looker Studio connected to BigQuery to visualize data nulls and write custom SQL conditional alerts that send webhook notifications to pipeline orchestrators.

Explanations

What Dataplex DataScan is

Dataplex DataScan is a fully managed, serverless governance capability in Google Cloud that automates data profiling and data quality validation directly across BigQuery tables and Cloud Storage data lakes without requiring data movement.

How it meets the requirements

Detect anomalies and outliers: Dataplex Data Profile scans (dataProfileSpec) compute column-level summary statistics, distinct values, null percentages, and distributions, allowing engineers to visually assess anomalies and data distributions directly within the Google Cloud console.
Automated continuous data quality validation: Dataplex Data Quality scans (dataQualitySpec) execute declarative rules, such as rangeExpectation, regexExpectation, setExpectation, and custom sqlAssertion checks against datasets on a defined schedule.
Export metrics and failed record queries: Post-scan actions (postScanActions.bigqueryExport.resultsTable) write comprehensive execution metadata—including rows_validated, success_percentage, failed_count, and a diagnostic failed_records_query SQL statement—directly into a target BigQuery table for historical auditing.
Threshold-based notifications: Configuring postScanActions.notificationReport.scoreThresholdTrigger triggers automated email alerts when the overall quality score drops below the specified tolerance.

Key advantages

Serverless execution: Runs directly using BigQuery compute resources without managing separate VM clusters or ETL runtimes.
Built-in diagnostics: Automatically generates SQL queries to extract the exact failed records for rapid troubleshooting.
Centralized governance: Integrates natively with the Dataplex Universal Catalog to publish data quality aspects alongside technical metadata.
Standardized metrics: Categorizes results across standardized data quality dimensions such as Completeness, Validity, and Uniqueness.

Why this is optimal

This approach provides an end-to-end, native Google Cloud framework for profiling, automated quality rule enforcement, auditing, and alert management, minimizing operational overhead while establishing automated data reliability.

View source documentation

Professional Cloud Data Engineer

Dataplex Data Profiling Quality Scans Monitoring

Professional Cloud Data Engineer

Dataplex Data Profiling Quality Scans Monitoring

What Dataplex DataScan is

How it meets the requirements

Key advantages

Why this is optimal

Related questions