Beam Pipeline Side Outputs Dead Letter Queue

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

A data engineering team operates a 24/7 streaming Cloud Dataflow pipeline written in Java that ingests high-volume JSON event streams from Pub/Sub and writes structured rows into BigQuery. Recently, unexpected schema changes upstream have caused malformed JSON payloads, leading to uncaught runtime parsing exceptions inside a DoFn that cause streaming worker tasks to retry indefinitely and stall the pipeline.

The team needs to update the Apache Beam pipeline architecture to validate incoming records, immediately divert uncorrectable corrupt payloads to a dead-letter storage sink for auditing and reprocessing, and ensure that valid data continues to flow to BigQuery without pipeline disruption.

Which design pattern should the team implement?

Implement a multi-output DoFn using TupleTag instances, enclose parsing logic in try-catch blocks, emit valid records to the main output tag, and emit unprocessable raw payloads wrapped in an error object to a dead-letter side output tag routed to an error sink

Configure the Dataflow execution parameters to disable worker retries, allowing uncaught UserCodeException errors to automatically drop corrupt records and write stack traces to Cloud Logging

Use a Flatten transform to duplicate the input PCollection into two parallel streams, running a strict parser on the primary stream and a custom error logger on the duplicate stream

Insert a Filter transform before the parsing DoFn that uses a regex pattern to drop invalid JSON strings and writes the dropped strings into an Apache Beam side input view

Professional Cloud Data Engineer

Beam Pipeline Side Outputs Dead Letter Queue

Related questions