Troubleshoot Dataflow Streaming Transient Persistent Errors

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

An enterprise runs a Dataflow streaming pipeline that ingests audit logs from Pub/Sub, applies in-flight transformations using a JavaScript User-Defined Function (UDF), and exports the events to an external monitoring endpoint (Splunk HEC).

During a system alert triggered by high backlog on the unprocessed dead-letter subscription, a data engineer analyzes Cloud Logging and error attributes attached to failed records:

Several worker log entries display Splunk write status code: 503 and Read timed out.
Messages in the unprocessed dead-letter topic have errorMessage attributes showing Splunk write status code: 403 and ReferenceError: device_id is not defined.

How should the data engineer classify these errors and remediate the pipeline to ensure full data recovery?

Classify the 503 status codes and timeout errors as transient failures that the pipeline automatically retries with exponential backoff; classify the 403 status code and UDF ReferenceError as persistent failures routed to the unprocessed topic. Fix the authorization credentials and UDF syntax, and then execute a replay pipeline to reprocess the unprocessed messages.

Classify the errors as worker memory exhaustion (GC thrashing). Enable Streaming Engine, configure custom SLF4J logging libraries using Logback to suppress 4xx and 5xx logs, and purge the unprocessed dead-letter subscription to clear the backlog.

Classify all errors as transient failures caused by downstream backpressure. Increase the maximum worker count and machine types in Dataflow, and configure the pipeline to retry all dead-letter topic messages automatically by increasing the retry threshold.

Classify all errors as systemic failures caused by invalid service account roles. Grant the Dataflow worker service account the roles/pubsub.admin and roles/dataflow.admin roles, and restart the Dataflow streaming job with the --drain option.

Professional Cloud Data Engineer

Troubleshoot Dataflow Streaming Transient Persistent Errors

Related questions