Data Quality Rules Monitoring and Contracts

Implementing real-time and batch data quality rules means applying checks for completeness, consistency, and correctness at different points in the data lifecycle. Cloud Dataflow (opens in a new tab) enforces per-record checks as data flows through ingestion and preprocessing, while Knowledge Catalog and BigQuery enforce dataset-level checks before data is used for training or analytics. Together they defend against missing data, malformed data, and malicious data such as data poisoning.

1.1Dimensions of Data Quality for Completeness, Consistency, and Correctness

Data quality rules validate three dimensions of fitness for purpose. Completeness checks verify that required records, fields, and values are present and that no expected data is missing. Consistency checks verify that data types, formats, and ranges match the expected schema as data moves from sources through transforms to sinks. Correctness checks verify that values and distributions reflect reality, including shifted distributions or injected records that could manipulate model behavior.

Robust validation before training verifies formats, ranges, and distributions together rather than only one dimension. Format and range checks catch malformed or out-of-bound values early, while distribution checks catch subtle drift or poisoning that per-record checks miss. This combination protects the validity, unbiasedness, reliability, and integrity of training data.

1.2Real-Time Validation Inside Dataflow Pipelines

Cloud Dataflow is the service for validating streaming and batch data inside ingestion and preprocessing code. Data flows from one or more sources through pipeline transforms to one or more sinks, and validation logic implemented in Python checks data types, formats, and ranges as each record passes through. The pipeline validates and sanitizes inputs and outputs in code, so malformed records and manipulation attempts are blocked before they reach downstream storage or model training.

Real-time validation depends on continuous monitoring of data pipelines and machine learning (ML) pipelines. Cloud Logging captures real-time logs from pipeline code, Cloud Monitoring collects metrics related to pipeline behavior, and Cloud Audit Logs collects audit trails of API calls that created, modified, or read data. A governance layer monitors these data flows, transformations, and access events to audit access, detect unauthorized attempts, and track inconsistent behavior or unexpected transformation.

Streaming validation carries higher operational complexity than batch validation. Continuous deployment can directly launch a batch pipeline as a new Dataflow job, while a streaming pipeline requires a replace-or-update decision and may require coordinating multiple Dataflow jobs to avoid business disruption. Core processing for either mode happens in the region specified for the Dataflow workers, so validation logic runs where the workers run even when sources and sinks are in other regions.

1.3Batch and Governed Validation with Knowledge Catalog and BigQuery

Knowledge Catalog is the centralized governance platform for managing data quality checks and data profiles across data platforms. The platform centrally discovers, manages, monitors, and governs data and artificial intelligence (AI) artifacts, and it provides access to trusted users for exploration and processing. Its governed capabilities include:

• Manage data quality checks and data profiles
• Manage data lineage, including column-level lineage in BigQuery
• Manage data discovery, exploration, and processing across data marts
• Manage feature metadata and model artifacts
• Create a business glossary to standardize vocabulary
• Enrich metadata with context through aspects and aspect types
• Unify governance across BigLake and open-format tables such as Iceberg and Delta

Automatic data quality in Knowledge Catalog verifies integrity at scale without hand-writing every rule. BigQuery complements those checks with anomaly detection using statistical methods or ML models, which detects anomalies in training data that might indicate data poisoning. Vertex AI Feature Store extends batch monitoring to features and feature groups by scheduling monitoring that tracks shifts after ingestion.

1.4Trade-Offs Between Streaming Validation and Batch Validation

Streaming validation minimizes the time between bad data arriving and detection, while batch validation maximizes the context available for judging correctness. Streaming checks see one record or window at a time, so they excel at completeness and consistency but cannot judge a full distribution. Batch checks see the whole table, profile, or feature set, so they excel at distribution, drift, and poisoning detection but act only after data has landed.

Choose streaming validation when the business requirement is immediate containment of malformed or malicious events before they propagate to sinks. Choose batch validation when the requirement is fitness for training or reporting, where format, range, and distribution must be verified together and lineage must be recorded for audit and root-cause analysis.

Designing validation pipelines for anomaly detection and monitoring means building automated systems that continuously check data fidelity from ingestion through transformation to serving. The pipeline captures logs and metrics into Cloud Logging and Cloud Monitoring, runs quality and statistical checks to detect drift and outliers, and routes findings through alerting into remediation. This closed loop maintains reliability because every stage from collection to response depends on shared telemetry, versioned results, and defined ownership.

2.1Centralized Telemetry with Logging, Monitoring, and Audit Logs

Centralized telemetry gives a validation pipeline its eyes and ears in production. Cloud Logging captures real-time event logs from data pipelines, including file arrivals, transformation errors, and access activity, while Cloud Monitoring collects numerical metrics such as row counts, null rates, latency, throughput, and custom data-quality metrics. Cloud Audit Logs (opens in a new tab) adds an API audit trail that records who called which service to create, modify, or read data, with Admin Activity logs enabled by default and Data Access logs enabled for services such as BigQuery.

Telemetry flows from pipeline components to Cloud Logging and Cloud Monitoring, where log-based metrics convert selected log events into monitorable time series for dashboards and alerts. Do not write personally identifiable information (PII) or confidential values into logs or experiment outputs, and export selected logs to BigQuery for long-term analysis or to a security information and event management (SIEM) system for centralized review.

2.2Automated Data Quality Checks and Drift Detection

Automated data quality checks validate that incoming and transformed data matches expected formats, ranges, distributions, and profiles before errors propagate downstream. Knowledge Catalog provides the governance layer that centrally discovers data, manages data lineage including column-level lineage in BigQuery, and runs automatic data-quality checks and data profiles. TensorFlow Data Validation (TFDV) validates ingestion and preprocessing logic and monitors data distributions over time, so shifts in feature distributions that signal data drift become visible as trends in Cloud Monitoring dashboards.

When drift exceeds a threshold, the monitoring signal can automatically trigger a retraining pipeline, while validation results flow into BigQuery for historical tracking and validation artifacts are archived in Cloud Storage. The three validation engines divide work by scale and programmability:

Use the engines together because a pipeline often depends on all three: Dataflow validates records in flight, BigQuery validates landed tables at scale, and Knowledge Catalog tracks quality history and lineage for audit and root-cause analysis.

2.3Anomaly and Outlier Detection in Production Data

Anomaly and outlier detection finds individual records or patterns that deviate from expected behavior, including sudden volume spikes, distribution shifts, and signs of data poisoning. Data poisoning is an attack in which malicious rows are injected into training data to manipulate model behavior or degrade performance, so the validation pipeline must treat unexpected access, transformation changes, and statistical outliers as security-relevant signals. BigQuery anomaly detection applies statistical methods or machine learning models directly to training and serving tables to flag suspicious deviations without moving data to a separate system.

Agent Platform Feature Store extends this monitoring to scheduled checks of features and feature groups, and model monitoring tracks model and data drift metrics such as prediction distribution shifts alongside infrastructure signals like latency and throughput.

2.4Alerting and Remediation Workflows from Detection to Response

Alerting and remediation workflows convert detections into reliable action with clear ownership and order of operations. The workflow moves telemetry from detection to response in sequence:

1. Create measures: Log events flow from Cloud Logging into log-based metrics, and pipeline code emits custom metrics to Cloud Monitoring for drift scores, anomaly counts, and validation failure rates.
2. Define alerting policies: Alerting policies in Cloud Monitoring evaluate the metrics against thresholds, service-level objectives (SLOs), and error-budget burn rates that provide early warning of reliability loss.
3. Route notifications: Firing alerts flow from the alerting policy to notification channels that reach the owning data or on-call team.
4. Execute remediation: The notified workflow triggers containment and repair, such as quarantining bad partitions, rolling back a load, replacing faulty nodes, adding capacity, or launching continuous retraining on updated data.

Remediation depends on preserved evidence and lineage to succeed. Validation results stored in BigQuery provide the queryable history that links an alert to the failing batch, while lineage in Knowledge Catalog traces the artifact from source dataset through Dataflow transforms to the served table for debugging. File-level integrity checks using Cloud Storage size and checksum validation, often run by Cloud Run functions on upload events or inside Dataflow, detect corruption or incomplete transfers before downstream checks run.

A data contract is a structured, enforceable agreement between data providers and consumers that defines schema, types, business rules, and allowed use. In Google Cloud, the contract is constructed in BigQuery schemas and Pub/Sub topics and made executable with Data Catalog tags, policy tables, and automated checks that validate, enforce, and audit compliance across pipelines.

3.1BigQuery Schemas and Table-Level Contract Rules

BigQuery (opens in a new tab) schemas define the structural half of a data contract through datasets, tables, columns, data types, and storage location. When a dataset is created the location is fixed as regional, such as us-central1, or multi-region, such as US or EU, which controls where data resides and which failure domain applies for availability planning. Because warehouses are often terabytes to petabytes and append-only with static historic data, contracts must also define whether new data is validated incrementally or the full table is revalidated, balancing recovery time and cost.

The Cloud Data Quality Engine runs contract rules against specified table columns and measures correctness and completeness as success percentages against thresholds. Results are written back as Data Catalog tags on the asset, so downstream pipelines and consumers can read contract status from metadata without rescanning the table.

3.2Pub/Sub Schemas and Event-Level Validation

Pub/Sub topics and subscriptions enforce the event half of a data contract as data flows from publishers to subscribers and then to processing. Publishers send messages to a predefined topic, subscribers pull through independent subscriptions, and a subscription can buffer backed-up events when pipelines stop so no contracted events are lost before processing resumes. This buffering gives low recovery point objective, but throughput quotas apply at the project level, so pipelines in separate projects with separate service accounts receive independent publisher and subscriber throughput.

Schema control at ingestion also limits which schemas, tables, and columns are streamed and which are excluded, preventing sensitive or out-of-contract fields from entering the destination. When automated checks detect encryption, de-identification, retention, or quality deviations, findings flow from the validator to operators through Pub/Sub, making Pub/Sub both the data plane for events and the alert plane for contract violations.

3.3Data Catalog Tags as Contract Metadata

Data Catalog tags, in Knowledge Catalog, make the contract discoverable, standardized, and machine-enforceable by attaching provider rules and measured status directly to each asset. Tag Engine applies tags automatically based on data sensitivity and a policy table stored in BigQuery, so business and governance rules defined once in a table propagate consistently to many assets. Built-in lineage then tracks provenance from original source through transforms, while ingestion scripts tag the source system to preserve traceability.

Contract tags cover distinct dimensions that together define fitness for purpose:

• provider sharing-agreement information that controls allowed consumption purposes
• encryption and de-identification techniques applied to the asset
• sensitivity, subject location, and assessment links for privacy impact assessments
• retention period and expiration action, such as archive or purge
• data quality results, including success percentages and thresholds for correctness and completeness
• cost metrics derived from Cloud Billing exports, such as query, storage, and egress cost
• provenance and original source identity plus column-level lineage where available

These tags do not themselves transform data; the tags store the contract state that BigQuery, Identity and Access Management (IAM), and automation then enforce.

3.4Contract Enforcement, Monitoring, and Audit Lifecycle

Contract enforcement, monitoring, and audit close the loop from definition to consequence through an ordered lifecycle where each stage depends on the prior stage. First, agreements, assessment requirements, and retention policies are defined in dedicated BigQuery tables. Then Tag Engine and ingestion logic apply the corresponding Data Catalog tags to assets. Then validators check reality against the tags, including automated encryption checks and Data Quality Engine column checks. Only after validation does enforcement act, followed by reporting and audit.

1. Define: provider and consumer sharing agreements, policy tables, and quality thresholds are stored in BigQuery.
2. Label: Data Catalog labels assets with agreement, security, privacy, retention, quality, and provenance tags.
3. Control access: consumer agreements are linked to IAM bindings, with row-level and column-level controls and least-privilege roles separating operational control from data access.
4. Enforce use: query labels require a valid consumption purpose that is validated against entitlements in BigQuery, and Record Manager purges or archives BigQuery tables based on retention tags.
5. Report and audit: discrepancies and quality deviations are published as findings using Pub/Sub, while a BigQuery audit trail records all access for compliance review.

The three layers divide responsibility by where they act:

Key Facts

• Cloud Dataflow validates per-record data types, formats, and ranges in flight, while Knowledge Catalog and BigQuery validate completeness, profiles, and distributions across the full dataset.
• Streaming validation enables faster containment of bad data but requires more complex replace-or-update operations, while batch validation provides full distribution context before training.
• Centralized telemetry flows from pipeline components to Cloud Logging for events, Cloud Monitoring for metrics and alerts, and Cloud Audit Logs for API audit trails.
• A data contract is enforced through BigQuery schemas and purpose checks, Pub/Sub topic schemas and buffered delivery, and Data Catalog tags that carry agreement, security, retention, and quality status.