A data engineering team is implementing a data cleansing and quality governance framework for raw transaction data ingested daily into BigQuery. To prevent data corruption from impacting downstream analytics, the team must meet the following requirements: Run visual data profiling on datasets to detect column distributions, null counts, anomalies, and outliers. Define and automate continuous data quality validation rules (including numeric range checks, regex pattern matches, and custom SQL assertions). Export detailed execution metrics, rule evaluation statuses, and queries for failed records to BigQuery for auditing. Automatically trigger notifications if data quality scores fall below a defined threshold. Which solution should the team implement in Dataplex to satisfy these requirements?

Configure Dataplex DataScan jobs to execute Data Profile scans for distribution insights, along with Data Quality scans utilizing dataQualitySpec rules, BigQuery post-scan export, and notification score thresholds.: ### What Dataplex DataScan is **Dataplex DataScan** is a fully managed, serverless governance capability in Google Cloud that automates **data profiling** and **data quality validation** directly across BigQuery tables and Cloud Storage data lakes without requiring data movement. ### How it meets the requirements - **Detect anomalies and outliers**: Dataplex **Data Profile scans** (`dataProfileSpec`) compute column-level summary statistics, distinct values, null percentages, and distributions, allowing engineers to visually assess anomalies and data distributions directly within the Google Cloud console. - **Automated continuous data quality validation**: Dataplex **Data Quality scans** (`dataQualitySpec`) execute declarative rules, such as `rangeExpectation`, `regexExpectation`, `setExpectation`, and custom `sqlAssertion` checks against datasets on a defined schedule. - **Export metrics and failed record queries**: Post-scan actions (`postScanActions.bigqueryExport.resultsTable`) write comprehensive execution metadata—including `rows_validated`, `success_percentage`, `failed_count`, and a diagnostic `failed_records_query` SQL statement—directly into a target BigQuery table for historical auditing. - **Threshold-based notifications**: Configuring `postScanActions.notificationReport.scoreThresholdTrigger` triggers automated email alerts when the overall quality score drops below the specified tolerance. ### Key advantages - **Serverless execution**: Runs directly using BigQuery compute resources without managing separate VM clusters or ETL runtimes. - **Built-in diagnostics**: Automatically generates SQL queries to extract the exact failed records for rapid troubleshooting. - **Centralized governance**: Integrates natively with the **Dataplex Universal Catalog** to publish data quality aspects alongside technical metadata. - **Standardized metrics**: Categorizes results across standardized data quality dimensions such as **Completeness**, **Validity**, and **Uniqueness**. ### Why this is optimal This approach provides an end-to-end, native Google Cloud framework for profiling, automated quality rule enforcement, auditing, and alert management, minimizing operational overhead while establishing automated data reliability.

Dataplex Data Profiling Quality Scans Monitoring | professional-cloud-data-engineer

Dataplex Data Profiling Quality Scans Monitoring

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A data engineering team is implementing a data cleansing and quality governance framework for raw transaction data ingested daily into BigQuery. To prevent data corruption from impacting downstream analytics, the team must meet the following requirements:

Run visual data profiling on datasets to detect column distributions, null counts, anomalies, and outliers.
Define and automate continuous data quality validation rules (including numeric range checks, regex pattern matches, and custom SQL assertions).
Export detailed execution metrics, rule evaluation statuses, and queries for failed records to BigQuery for auditing.
Automatically trigger notifications if data quality scores fall below a defined threshold.

Which solution should the team implement in Dataplex to satisfy these requirements?

Use Sensitive Data Protection (DLP) discovery scans to profile dataset columns, and configure Security Command Center finding posture policies to enforce data quality ranges.

Configure Dataplex DataScan jobs to execute Data Profile scans for distribution insights, along with Data Quality scans utilizing dataQualitySpec rules, BigQuery post-scan export, and notification score thresholds.

Execute BigQuery ML data validation functions (ML.DESCRIBE_DATA and ML.VALIDATE_DATA_DRIFT) via scheduled queries, logging error counts into custom audit tables.

Deploy Looker Studio connected to BigQuery to visualize data nulls and write custom SQL conditional alerts that send webhook notifications to pipeline orchestrators.

Professional Cloud Data Engineer

Dataplex Data Profiling Quality Scans Monitoring

Related questions