Data validation

Construct and Enforce Data Contracts and Schemas

Data Contracts with Pub/Sub Schemas

A data contract is a formal agreement between data producers and consumers that defines the structure and format of exchanged data. In Google Cloud, data producers enforce these contracts by assigning a Pub/Sub schema in Avro or Protobuf format directly to a message topic. The schema is validated at the ingestion point, so any incoming payload that does not match the schema is rejected before it reaches downstream pipelines. This automatic validation prevents malformed data from causing failures later in the processing chain.

Schema Versioning Strategies

When a data contract must change, the producer and consumer need to coordinate the update. Three schema versioning strategies are available:

• New Topic: Create a new Pub/Sub topic for every structural change. This guarantees that each topic contains only messages with a single, unvarying schema, making the contract strongly typed.
• Single Topic: Use a single topic and include a message attribute that indicates the schema version. All data flows through one topic, but consumers must inspect the attribute to apply the correct schema.
• Hybrid Approach: Add an arbitrary data section to the schema. This balances strong validation for the fixed part of the schema with flexibility for fields that change over time.

Choosing the right strategy depends on how much pipeline downtime is acceptable during migrations and how tightly the producer and consumer must coordinate.

Automated Data Quality Monitoring

Beyond format enforcement, data contracts must also ensure that the content meets business rules. The Cloud Data Quality Engine runs automated checks against BigQuery columns to measure dimensions such as completeness (are fields missing?) and correctness (do values fall within allowed ranges?). The resulting quality scores are stored as metadata tags in Dataplex Universal Catalog, making them searchable and transparent for consumers. This integration turns validation from a one-time check into a continuous monitoring process.

BigQuery Storage Write API and Poison Records

When streaming validated records into BigQuery, the BigQuery Storage Write API enforces the target schema and provides transactional integrity. Any record that violates the schema is classified as a poison record and is redirected to a dead letter queue for isolated troubleshooting. The API supports committed-type streams with client-provided offsets, which guarantee exactly-once semantics. This means that if a validation attempt fails and is retried, the record is not duplicated in the table.

Design Validation Pipelines for Anomaly Detection and Monitoring

Automated Anomaly Detection Systems

Automated anomaly detection identifies data drift, outliers, and unexpected patterns in production data pipelines. Cloud Monitoring and Cloud Logging are the foundational services for collecting metrics and tracking system behavior. These services integrate with custom metrics to give teams granular visibility into pipeline health. Google Cloud provides several specialized tools for anomaly detection:

• Vertex AI Model Monitoring detects data drift and identifies anomalies in machine learning model inputs and outputs.
• TensorFlow Data Validation (TFDV) validates schemas and detects statistical anomalies in datasets.
• Automated data quality scans in BigQuery continuously validate tables as data arrives.

These tools together allow teams to catch issues before they affect downstream applications.

Alerting and Remediation Workflows

When an anomaly or threshold violation is detected, the system must notify the right people and trigger corrective actions. Organizations configure Cloud Monitoring alerts to fire when custom metrics exceed defined thresholds. Each alerting policy specifies the condition, the notification channel (such as email or Slack), and optionally an automated remediation workflow. Remediation workflows can run without manual intervention, for example by restarting a failed pipeline or quarantining bad records. Regularly reviewing and updating these policies ensures they remain effective as the system evolves.

Observability and Reliability Practices

Holistic observability means monitoring infrastructure, application code, data, and model behavior to proactively detect and diagnose issues. Teams should adopt Site Reliability Engineering (SRE) principles and define reliability goals that align with business objectives. To achieve comprehensive observability, teams should:

• Monitor logs for suspicious activity, unauthorized access attempts, or unexpected modifications to critical data.
• Export logs to BigQuery for cost-effective long-term storage and retrospective analysis.
• Integrate with Security Command Center for centralized security monitoring.

These practices ensure that validation pipelines are not just one-time checks but part of a continuous reliability framework.

Implement Real-Time and Batch Data Quality Rules

Real-Time Validation with Dataflow

When data must be validated immediately—for example, in fraud detection or live dashboards—real-time validation is required. Google Cloud Dataflow is the primary tool for building streaming pipelines that apply validation rules as data arrives. The pipeline checks each event for anomalies, missing fields, or format inconsistencies as it streams from sources like Pub/Sub. Real-time validation demands more resources and careful pipeline design to handle high-velocity data without introducing significant latency.

Batch Validation with Dataform and Dataplex

When immediate processing is not needed, batch validation offers a more efficient and cost-effective approach. Dataform allows teams to define and schedule SQL-based data quality checks that run on large datasets at rest, such as those in BigQuery. Dataplex provides a unified governance platform where you can configure and automate data quality scans across data lakes and warehouses. Batch validation is ideal for comprehensive audits, historical data analysis, and ensuring data integrity after large ingestion jobs, because it can process vast volumes of data during off-peak hours.

Designing a Comprehensive Strategy

A robust data quality system often combines both real-time and batch validation. Use real-time validation to filter out obviously malformed records at the point of ingestion, ensuring only clean data enters the system. Then apply more complex, resource-intensive batch validation rules periodically to uncover deeper integrity issues, such as referential consistency or business logic violations across historical datasets. This layered strategy balances the need for speed with the demand for thoroughness.

Key Considerations and Trade-offs

The choice between real-time and batch validation depends on business requirements. Real-time validation provides immediate data quality signals but at a higher operational cost and complexity. Batch validation is generally more economical and easier to manage for large-scale checks but introduces a delay in identifying issues. The trade-off is between lower latency and lower cost. Designers must select the right tool—Dataflow for streaming, Dataform for SQL transformations, or Dataplex for automated governance—and the right processing model to ensure data remains accurate and fit for purpose throughout its lifecycle.

Key Facts

• Data contracts enforce structured schemas at the ingestion point using Pub/Sub schemas in Avro or Protobuf format, preventing malformed data from reaching downstream pipelines.
• Schema versioning strategies include New Topic (each version gets its own topic), Single Topic (version in a message attribute), and Hybrid Approach (arbitrary data section for flexibility).
• The Cloud Data Quality Engine runs automated checks on BigQuery columns and stores results as metadata tags in Dataplex Universal Catalog.
• The BigQuery Storage Write API with committed-type streams and client-provided offsets provides exactly-once semantics and redirects poison records to a dead letter queue.
• Real-time validation with Dataflow provides immediate feedback at higher cost, while batch validation with Dataform or Dataplex is more economical for large-scale, periodic checks.