professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
An enterprise is deploying an Apache Beam streaming pipeline on Google Cloud Dataflow to ingest high-throughput real-time telemetry from a newly integrated partner source. During integration testing, occasional malformed payloads and invalid schema records cause worker exceptions, which lead to indefinite bundle retries, delayed watermarks, and pipeline processing stalls.
The data engineering team must ensure pipeline reliability, prevent malformed records from halting stream execution, preserve failed messages with diagnostic context for post-mortem replay, and establish automated operational alerting.
Which error-handling and observability architecture should the team implement?
The dead-letter pattern (or error side-output routing) combined with Cloud Monitoring backlog metrics is the recommended architecture for robust stream processing in Google Cloud Dataflow. Instead of allowing unhandled exceptions to crash worker tasks, the pipeline catches data errors, encapsulates the bad payload with contextual diagnostic information (such as error message and timestamp), and routes it to an auxiliary storage destination or dead-letter Pub/Sub topic.
StudyCerts
StudyCertsPCollectionTuple (dead-letter topic), valid streaming traffic continues uninterrupted without stalling the watermark.errorMessage, failure timestamp, delivery attempt counts), facilitating root-cause analysis and automated replay once issues are resolved.pubsub.googleapis.com/subscription/num_undelivered_messages metric and trigger automated SRE alerts when message failure counts exceed acceptable operational thresholds.Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.