Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An enterprise is deploying an Apache Beam streaming pipeline on Google Cloud Dataflow to ingest high-throughput real-time telemetry from a newly integrated partner source. During integration testing, occasional malformed payloads and invalid schema records cause worker exceptions, which lead to indefinite bundle retries, delayed watermarks, and pipeline processing stalls.
The data engineering team must ensure pipeline reliability, prevent malformed records from halting stream execution, preserve failed messages with diagnostic context for post-mortem replay, and establish automated operational alerting.
Which error-handling and observability architecture should the team implement?
This approach writes intermediate PCollections out to Cloud Storage buckets between transformation steps to checkpoint data and uses Cloud Functions to detect schema discrepancies.
You were designing an expensive, non-deterministic batch processing pipeline where saving intermediate state prevents re-running long-duration batch computations.
The dead-letter pattern (or error side-output routing) combined with Cloud Monitoring backlog metrics is the recommended architecture for robust stream processing in Google Cloud Dataflow. Instead of allowing unhandled exceptions to crash worker tasks, the pipeline catches data errors, encapsulates the bad payload with contextual diagnostic information (such as error message and timestamp), and routes it to an auxiliary storage destination or dead-letter Pub/Sub topic.
PCollectionTuple (dead-letter topic), valid streaming traffic continues uninterrupted without stalling the watermark.errorMessage, failure timestamp, delivery attempt counts), facilitating root-cause analysis and automated replay once issues are resolved.pubsub.googleapis.com/subscription/num_undelivered_messages metric and trigger automated SRE alerts when message failure counts exceed acceptable operational thresholds.This option deploys the Monitoring agent (--experiments=enable_stackdriver_agent_metrics) to capture infrastructure-level VM metrics such as CPU, memory, and disk usage on Dataflow worker instances.
The primary requirement was to monitor compute infrastructure constraints or worker virtual machine resource saturation.
This option relies on configuring worker retry limits to discard failing records and setting up Cloud Logging alerts for fatal worker exceptions.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated