Dataflow Ai Enrichment Quota Batching Error Handling

Worked example. The correct answer is already marked and every option is explained below, so there is nothing to select here. To answer questions yourself, start the free trial.

Your team is building a real-time streaming Dataflow pipeline that processes incoming customer feedback from Pub/Sub. The pipeline must enrich the unstructured text and attached images using pre-trained Google Cloud APIs, including the Cloud Translation API and Cloud Vision API, before writing structured records to BigQuery.

During peak traffic events, the pipeline experiences throughput degradation, high processing lag, and failed execution cycles caused by 429 (Rate Limit Exceeded) quota errors from the external AI APIs.

Which design pattern should you implement in the Dataflow pipeline to manage API quotas, handle transient errors gracefully, and maximize throughput?

Increase the maximum number of Dataflow worker instances and disable worker autoscaling, while immediately dropping failed API requests to avoid retry bottlenecks.

Write all streaming records directly to Managed Lustre storage, and schedule a recurring Cloud Composer DAG to trigger batch API enrichment jobs.

Batch input elements using GroupIntoBatches, initialize API clients in the setup method of the DoFn, implement exponential backoff with jitter for transient retries, and route non-retryable errors to a dead-letter side output.

Use an upstream Cloud Function triggered on every Pub/Sub message to synchronously call the AI APIs and append predictions before ingestion by Dataflow.

Professional Cloud Data Engineer

Dataflow Ai Enrichment Quota Batching Error Handling

Related questions