Dataflow Streaming Validation Dead Letter Architecture

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An e-commerce company ingests high-throughput user clickstream events via Cloud Pub/Sub and processes them using a streaming Dataflow pipeline before persisting the records into BigQuery. Upstream mobile application updates have occasionally introduced unannounced schema changes, unexpected null values in mandatory telemetry attributes, and malformed JSON payloads, causing unhandled pipeline exceptions and data corruption in downstream reporting tables.

The data engineering team needs to implement an automated validation framework that detects schema drift and corrupted records in real time, isolates non-conforming payloads without interrupting the processing of valid events, and alerts engineers when anomaly rates exceed predefined thresholds.

Which architecture should the team deploy?

Enable schema auto-detection and ignore unknown values on the BigQuery streaming sink within the Dataflow pipeline to coerce missing and drifted fields automatically

Configure an Cloud Monitoring alert on Dataflow system lag that automatically triggers a Cloud Function to stop the streaming job when lag increases

Deploy a Cloud Function subscribed to Pub/Sub to call the Cloud Data Loss Prevention (Cloud DLP) API for schema structural validation before forwarding records to Dataflow

Implement schema validation within a Dataflow DoFn using dead-letter side outputs for malformed or non-conforming records, increment custom pipeline metrics, and emit alerts via Cloud Monitoring

Professional Cloud Data Engineer

Dataflow Streaming Validation Dead Letter Architecture

Related questions