Dataflow Streaming Error Handling And Monitoring

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise is deploying an Apache Beam streaming pipeline on Google Cloud Dataflow to ingest high-throughput real-time telemetry from a newly integrated partner source. During integration testing, occasional malformed payloads and invalid schema records cause worker exceptions, which lead to indefinite bundle retries, delayed watermarks, and pipeline processing stalls.

The data engineering team must ensure pipeline reliability, prevent malformed records from halting stream execution, preserve failed messages with diagnostic context for post-mortem replay, and establish automated operational alerting.

Which error-handling and observability architecture should the team implement?

Enable the Cloud Monitoring Ops Agent on Dataflow worker VMs and set an alert policy based on the worker CPU utilization metric exceeding 90%

Implement try-catch validation in Beam transforms to route malformed elements to a dead-letter Pub/Sub topic with error metadata, and configure a Cloud Monitoring alert on the dead-letter subscription's unacknowledged message backlog

Configure Dataflow's maximum retry count parameter to 4 on worker transforms, discard failed elements automatically, and configure log-based alerts in Cloud Logging for worker fatal logs

Materialize intermediate PCollections into Cloud Storage after every transform step and configure a Cloud Function on bucket object creation to scan for errors

Professional Cloud Data Engineer

Dataflow Streaming Error Handling And Monitoring

Related questions