Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A data engineering team operates a 24/7 streaming Cloud Dataflow pipeline written in Java that ingests high-volume JSON event streams from Pub/Sub and writes structured rows into BigQuery. Recently, unexpected schema changes upstream have caused malformed JSON payloads, leading to uncaught runtime parsing exceptions inside a DoFn that cause streaming worker tasks to retry indefinitely and stall the pipeline.
The team needs to update the Apache Beam pipeline architecture to validate incoming records, immediately divert uncorrectable corrupt payloads to a dead-letter storage sink for auditing and reprocessing, and ensure that valid data continues to flow to BigQuery without pipeline disruption.
Which design pattern should the team implement?
Filter transform before the parsing DoFn that uses a regex pattern to drop invalid JSON strings and writes the dropped strings into an Apache Beam side input viewA Filter transform discards elements that do not satisfy a boolean predicate, while side inputs provide additional read-only auxiliary datasets (such as configuration lookup tables or singleton views) to a ParDo transform.
Filter transform simply drops elements from the PCollection without a native mechanism to route rejected records to a separate downstream dead-letter destination.DoFn to read external data alongside the main input element, not collect emitted errors.Gson or JsonToRow.You needed to enrich the main data stream with a slowly changing lookup table or broadcast dictionary loaded from Cloud Storage or BigQuery.
DoFn using TupleTag instances, enclose parsing logic in try-catch blocks, emit valid records to the main output tag, and emit unprocessable raw payloads wrapped in an error object to a dead-letter side output tag routed to an error sinkIn Apache Beam, a multi-output transform allows a single ParDo (or DoFn) to produce multiple distinct PCollection streams differentiated by TupleTag identifiers. This is the standard architectural pattern for building dead-letter queues (DLQs) and robust error-handling pipelines in both batch and streaming execution on Cloud Dataflow.
try-catch blocks within the DoFn.ProcessElement method, unparseable elements do not raise unhandled runtime exceptions. This prevents the Dataflow streaming runner from entering infinite retry loops.MultiOutputReceiver.get(deadLetterTag).output(errorObj).Row structures and emitted to the primary output TupleTag, allowing downstream BigQuery insertion transforms to proceed uninterrupted.Unlike throwing exceptions or dropping corrupt data silently, using Beam side outputs (TupleTag) allows branching the execution graph cleanly into distinct downstream pipelines—one for production analytical storage and another for dead-letter archival in Cloud Storage, Pub/Sub, or BigQuery.
UserCodeException errors to automatically drop corrupt records and write stack traces to Cloud LoggingThis approach attempts to handle errors by suppressing runner-level retry mechanisms and relying on Cloud Logging output instead of implementing structured data handling within the Beam pipeline.
PCollection that can be automatically audited, monitored, or reprocessed.You were simply looking to capture ad-hoc diagnostic errors during local development or unit testing on a DirectRunner where crashing on invalid input is acceptable.
Flatten transform to duplicate the input PCollection into two parallel streams, running a strict parser on the primary stream and a custom error logger on the duplicate streamThe Flatten transform in Apache Beam takes multiple PCollection objects of the same type and merges them into a single logical PCollection. It is not used to duplicate or split streams.
Flatten merges collections together rather than splitting or duplicating a collection into parallel branches.DoFn would still encounter the malformed JSON records and throw unhandled runtime exceptions, causing worker tasks to retry indefinitely and stalling the main branch.Flatten merges $N$ collections of type T into 1 collection of type T.You had multiple separate source topics or input files that all shared identical data types and needed to be unified into a single stream before validation.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated