professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
An e-commerce company ingests high-throughput user clickstream events via Cloud Pub/Sub and processes them using a streaming Dataflow pipeline before persisting the records into BigQuery. Upstream mobile application updates have occasionally introduced unannounced schema changes, unexpected null values in mandatory telemetry attributes, and malformed JSON payloads, causing unhandled pipeline exceptions and data corruption in downstream reporting tables.
The data engineering team needs to implement an automated validation framework that detects schema drift and corrupted records in real time, isolates non-conforming payloads without interrupting the processing of valid events, and alerts engineers when anomaly rates exceed predefined thresholds.
Which architecture should the team deploy?
The dead-letter pattern with side outputs is an architectural best practice in Apache Beam and Google Cloud Dataflow for building fault-tolerant streaming pipelines. In this pattern, incoming payloads are parsed and validated inside a DoFn. Valid records are emitted to the standard main PCollection for downstream transformations and storage, while invalid, malformed, or schema-deviant records are caught and routed to a side output PCollection (the dead-letter queue) without halting pipeline execution.
StudyCerts
StudyCertsDoFn explicitly parses the JSON payloads against expected schema models, detecting missing mandatory fields, incorrect data types, or unrecognized attributes as each event arrives.NullPointerException or serialization errors.This solution isolates corrupt data at the ingestion stage, protecting downstream analytical data warehouses while maintaining continuous stream availability.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.