Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
A financial analytics company processes millions of real-time point-of-sale transactions using an Apache Beam pipeline running on Google Cloud Dataflow. The pipeline ingests transactions from Cloud Pub/Sub and calculates aggregated financial metrics per merchant across 1-hour event-time windows.
The engineering team has identified the following requirements:
Which pipeline design pattern should the team implement?
This option replaces fixed 1-hour calendar windows with session windows defined by a 2-hour inactivity gap duration, relying on default watermark tracking and worker storage scaling to manage state.
allowedLateness and late trigger firings, session windows merge solely based on on-time watermark arrival, failing to emit intermediate updates when delayed transactions arrive.The requirement was to track continuous user browsing activity where a session is considered closed only after 2 hours of complete merchant inactivity.
.withAllowedLateness(Duration.standardHours(2)) and an AfterWatermark trigger emitting late firings with accumulatingFiredPanes(). Use a side output on a subsequent DoFn to capture elements arriving after the window expiration threshold and route them to Cloud Storage.In Apache Beam and Google Cloud Dataflow, managing late-arriving data requires orchestrating event-time windowing, watermark tracking, allowed lateness, and triggers. This solution configures 1-hour fixed windows with an explicit allowed lateness of 2 hours, enables late pane emissions via triggers, and routes data that arrives past the allowed lateness threshold to a dead-letter side output.
.withAllowedLateness(Duration.standardHours(2)) instructs Dataflow to keep the window state active for 2 hours after the watermark passes the end of the window. When late data arrives, the AfterWatermark.pastEndOfWindow() trigger fires late panes with accumulatingFiredPanes(), producing updated aggregate totals for the merchant.window_end + allowed_lateness), Dataflow automatically closes and garbage-collects the window state, bounding state memory and preventing resource exhaustion.DoFn with a TupleTag (or using Beam's dropped data side output), routing those elements to Cloud Storage for offline reprocessing.window_end + allowed_lateness, reclaiming state storage across Dataflow workers.accumulatingFiredPanes() ensures each late trigger emission reflects both on-time and late data.This approach directly balances computation accuracy, pipeline operational stability, and cost efficiency. It avoids unbounded state retention while providing a resilient audit trail for out-of-order stragglers.
.withAllowedLateness to unbounded using Duration.millis(Long.MAX_VALUE), and write all emitted panes to BigQuery with CDC deduplication.This option configures sliding windows based on processing time rather than event time, and sets an unbounded allowed lateness threshold (Long.MAX_VALUE) while relying on downstream deduplication in Google Cloud BigQuery.
allowedLateness to Long.MAX_VALUE forces Dataflow to retain internal state for every window indefinitely. Over time, persistent worker disk and memory usage will surge, causing out-of-memory crashes and pipeline failure.The workload required a moving average of recent transaction throughput calculated according to server arrival time, and storage cost or state size was not a constraint.
AfterProcessingTime trigger with discardingFiredPanes(), and configure a Pub/Sub dead-letter topic on the subscription to capture unparsed messages.This option applies standard 1-hour fixed windows without extending allowed lateness (leaving it at the default of zero), triggers pane evaluation on processing-time intervals with discardingFiredPanes(), and sets up a Cloud Pub/Sub dead-letter topic.
Duration.ZERO), any data arriving after the event-time watermark passes the end of the 1-hour window is immediately dropped and ignored by the aggregation step.discardingFiredPanes() discards previously accumulated state upon firing, which would emit only the delta of new data rather than the full updated merchant totals.The business requirement dictated that late data should be completely discarded to minimize latency, and the primary failure mode was malformed ingestion payloads in Cloud Pub/Sub.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated