Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
Your team is building a real-time streaming Dataflow pipeline that processes incoming customer feedback from Pub/Sub. The pipeline must enrich the unstructured text and attached images using pre-trained Google Cloud APIs, including the Cloud Translation API and Cloud Vision API, before writing structured records to BigQuery.
During peak traffic events, the pipeline experiences throughput degradation, high processing lag, and failed execution cycles caused by 429 (Rate Limit Exceeded) quota errors from the external AI APIs.
Which design pattern should you implement in the Dataflow pipeline to manage API quotas, handle transient errors gracefully, and maximize throughput?
This option converts a real-time streaming pipeline into a batch file-based workflow using a Managed Lustre parallel file system orchestrated by Cloud Composer (Apache Airflow).
You were training custom computer vision models requiring petabyte-scale high-throughput POSIX checkpointing on GPU clusters.
Streaming pipelines should process data continuously in-flight using Dataflow's unified stream engine rather than dumping streaming micro-events onto shared parallel storage.
This approach places a serverless Cloud Function in front of the Dataflow streaming pipeline to handle the enrichment step on a per-event basis.
The workload consisted of infrequent, asynchronous webhook notifications requiring light processing rather than a high-throughput, continuous streaming data stream.
Dataflow natively supports ML and AI component integration directly within pipeline transforms, eliminating the need to decouple streaming ETL workflows across standalone microservices.
This option attempts to solve processing lag by vertically and horizontally over-provisioning compute resources while discarding any data that fails during external API calls.
429 (Rate Limit Exceeded) errors.429 responses.The bottleneck were CPU-bound internal transformations that did not interact with external rate-limited APIs and where dropped messages were deemed acceptable.
Scaling up Dataflow compute nodes does not resolve external service quota limits; in fact, higher parallelism against constrained external APIs generally exacerbates rate-limiting issues.
GroupIntoBatches, initialize API clients in the setup method of the DoFn, implement exponential backoff with jitter for transient retries, and route non-retryable errors to a dead-letter side output.This architecture combines Apache Beam micro-batching transforms (GroupIntoBatches), lifecycle optimization (setup method in DoFn), client-side rate limiting/backoff, and dead-letter queuing (side outputs) to integrate pre-trained AI APIs cleanly into high-throughput Dataflow pipelines.
GroupIntoBatches groups incoming streaming elements into multi-item payloads, maximizing payload efficiency against API batch endpoints.429 (Rate Limit Exceeded) errors when burst limits are approached, allowing temporary quota windows to reset.setup() lifecycle method of the DoFn ensures that connection pools and authentication tokens are reused across multiple bundles on a worker rather than reinstantiated for every element.429 and 503 responses without crashing pipeline stages.This pattern directly addresses the root causes of streaming pipeline stalls when communicating with rate-limited external REST/gRPC endpoints. It optimizes compute and network efficiency while guaranteeing pipeline continuity under heavy load.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated