professional-cloud-data-engineer
Prepare and test your skills
Prepare and test your skills
Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.
Keep the momentum going with these hand-picked practice scenarios
Want more questions like this?
Get a free certification question every week.
Last updated
An enterprise runs a Dataflow streaming pipeline that ingests audit logs from Pub/Sub, applies in-flight transformations using a JavaScript User-Defined Function (UDF), and exports the events to an external monitoring endpoint (Splunk HEC).
During a system alert triggered by high backlog on the unprocessed dead-letter subscription, a data engineer analyzes Cloud Logging and error attributes attached to failed records:
StudyCerts
StudyCertsSplunk write status code: 503 and Read timed out.errorMessage attributes showing Splunk write status code: 403 and ReferenceError: device_id is not defined.How should the data engineer classify these errors and remediate the pipeline to ensure full data recovery?
Classify the errors as worker memory exhaustion (GC thrashing). Enable Streaming Engine, configure custom SLF4J logging libraries using Logback to suppress 4xx and 5xx logs, and purge the unprocessed dead-letter subscription to clear the backlog.
Classify all errors as transient failures caused by downstream backpressure. Increase the maximum worker count and machine types in Dataflow, and configure the pipeline to retry all dead-letter topic messages automatically by increasing the retry threshold.
Classify the 503 status codes and timeout errors as transient failures that the pipeline automatically retries with exponential backoff; classify the 403 status code and UDF ReferenceError as persistent failures routed to the unprocessed topic. Fix the authorization credentials and UDF syntax, and then execute a replay pipeline to reprocess the unprocessed messages.
Classify all errors as systemic failures caused by invalid service account roles. Grant the Dataflow worker service account the roles/pubsub.admin and roles/dataflow.admin roles, and restart the Dataflow streaming job with the --drain option.
Classify the errors as worker memory exhaustion (GC thrashing). Enable Streaming Engine, configure custom SLF4J logging libraries using Logback to suppress 4xx and 5xx logs, and purge the unprocessed dead-letter subscription to clear the backlog.
Classify all errors as transient failures caused by downstream backpressure. Increase the maximum worker count and machine types in Dataflow, and configure the pipeline to retry all dead-letter topic messages automatically by increasing the retry threshold.
Classify the 503 status codes and timeout errors as transient failures that the pipeline automatically retries with exponential backoff; classify the 403 status code and UDF ReferenceError as persistent failures routed to the unprocessed topic. Fix the authorization credentials and UDF syntax, and then execute a replay pipeline to reprocess the unprocessed messages.
In enterprise event streaming architectures using Cloud Dataflow and Pub/Sub, errors are categorized into transient errors (temporary server overload or network blips) and persistent/systemic errors (authentication failures or syntax/runtime bugs in transformation code). Transient errors can be resolved with automated retries, while persistent errors require manual code or configuration intervention followed by dead-letter replay.
503 Service Unavailable and network socket Read timed out errors occur due to temporary endpoint throttling or network latency spikes. The pipeline natively applies exponential backoff to re-deliver these without routing them to the dead-letter queue.403 Forbidden indicates an invalid or expired authentication token (HEC token), and ReferenceError: device_id is not defined indicates a breaking syntax or variable scope defect inside the JavaScript User-Defined Function (UDF). Because retrying identical payloads against these errors will never succeed, the pipeline forwards them to the unprocessed topic to prevent blocking pipeline execution.5xx errors are already handled automatically by the underlying runner.This solution correctly diagnoses both error types using structured log properties, avoids unnecessary intervention for self-healing transient errors, and implements the standard dead-letter replay operational pattern recommended for Google Cloud stream processing.
Classify all errors as systemic failures caused by invalid service account roles. Grant the Dataflow worker service account the roles/pubsub.admin and roles/dataflow.admin roles, and restart the Dataflow streaming job with the --drain option.