professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.
Last updated
A data engineering team operates a 24/7 streaming Cloud Dataflow pipeline written in Java that ingests high-volume JSON event streams from Pub/Sub and writes structured rows into BigQuery. Recently, unexpected schema changes upstream have caused malformed JSON payloads, leading to uncaught runtime parsing exceptions inside a DoFn that cause streaming worker tasks to retry indefinitely and stall the pipeline.
The team needs to update the Apache Beam pipeline architecture to validate incoming records, immediately divert uncorrectable corrupt payloads to a dead-letter storage sink for auditing and reprocessing, and ensure that valid data continues to flow to BigQuery without pipeline disruption.
Which design pattern should the team implement?
Configure the Dataflow execution parameters to disable worker retries, allowing uncaught UserCodeException errors to automatically drop corrupt records and write stack traces to Cloud Logging
Insert a Filter transform before the parsing DoFn that uses a regex pattern to drop invalid JSON strings and writes the dropped strings into an Apache Beam side input view
Implement a multi-output DoFn using TupleTag instances, enclose parsing logic in try-catch blocks, emit valid records to the main output tag, and emit unprocessable raw payloads wrapped in an error object to a dead-letter side output tag routed to an error sink
Use a Flatten transform to duplicate the input PCollection into two parallel streams, running a strict parser on the primary stream and a custom error logger on the duplicate stream
Configure the Dataflow execution parameters to disable worker retries, allowing uncaught UserCodeException errors to automatically drop corrupt records and write stack traces to Cloud Logging
Insert a Filter transform before the parsing DoFn that uses a regex pattern to drop invalid JSON strings and writes the dropped strings into an Apache Beam side input view
Implement a multi-output DoFn using TupleTag instances, enclose parsing logic in try-catch blocks, emit valid records to the main output tag, and emit unprocessable raw payloads wrapped in an error object to a dead-letter side output tag routed to an error sink
In Apache Beam, a multi-output transform allows a single ParDo (or DoFn) to produce multiple distinct PCollection streams differentiated by TupleTag identifiers. This is the standard architectural pattern for building dead-letter queues (DLQs) and robust error-handling pipelines in both batch and streaming execution on Cloud Dataflow.
StudyCerts
StudyCertstry-catch blocks within the DoFn.ProcessElement method, unparseable elements do not raise unhandled runtime exceptions. This prevents the Dataflow streaming runner from entering infinite retry loops.MultiOutputReceiver.get(deadLetterTag).output(errorObj).Row structures and emitted to the primary output TupleTag, allowing downstream BigQuery insertion transforms to proceed uninterrupted.Unlike throwing exceptions or dropping corrupt data silently, using Beam side outputs (TupleTag) allows branching the execution graph cleanly into distinct downstream pipelines—one for production analytical storage and another for dead-letter archival in Cloud Storage, Pub/Sub, or BigQuery.
Use a Flatten transform to duplicate the input PCollection into two parallel streams, running a strict parser on the primary stream and a custom error logger on the duplicate stream