Prepare and test your skills

Prepare and test your skills

Free sample question, answered and explained below. The Professional Data Engineer practice bank has 1,353 more like it. Try 10 free: instant score, no account.
An e-commerce company is building a real-time analytics streaming pipeline in Apache Beam on Cloud Dataflow to process high-volume clickstream and transaction events from Pub/Sub.
The pipeline must fulfill the following architectural requirements:
Which combination of Apache Beam transformations, windowing strategies, triggers, and enrichment mechanisms should you implement?
This configuration defines overlapping sliding windows (SlidingWindows) evaluated according to processing time, using stateful ValueState cells for metadata management.
SlidingWindows creates fixed-length overlapping time slices, resulting in duplicated element processing across multiple windows rather than activity-gap-based sessionization.AfterProcessingTime triggers evaluates worker system clocks rather than event timestamps, failing to manage event-time skew or handle late-arriving event bounds properly.ValueState stores per-key state in worker disk/memory, which is designed for tracking keyed historical metrics over time rather than broadcasting global slowly changing reference metadata across all keys.You needed continuous rolling moving averages (e.g., rolling 15-minute trends calculated every minute) indexed strictly by arrival time.
This solution uses Apache Beam's native Sessions windowing, custom trigger composition, accumulation modes, and PCollectionView side inputs to build an adaptive, low-latency streaming pipeline.
Sessions.withGapDuration(Duration.standardMinutes(15)) defines data-driven windows that group user activity per key and close only after a 15-minute period of inactivity.AfterWatermark.pastEndOfWindow() trigger combined with .withEarlyFirings(...) and .withLateFirings(...) allows the pipeline to emit speculative results ahead of watermark completion and emit refined results when late data arrives..withAllowedLateness(Duration.standardHours(1)) retains window state for up to an hour after the watermark passes. Using accumulatingFiredPanes() ensures that subsequent late pane firings include previous data to provide updated cumulative totals.PCollectionView side input into a ParDo allows worker threads to perform in-memory lookups for store metadata against streaming elements without blocking stream consumption.This approach directly satisfies all session boundaries, handling of late-arriving data, and reference data enrichment requirements natively within the Beam execution model.
This option applies fixed time-interval tumbling windows (FixedWindows), uses the default watermark trigger, discards previously fired panes, and calls external services synchronously inside a ParDo function.
FixedWindows slices data into strict, calendar-aligned intervals (e.g., [12:00–12:15), [12:15–12:30)) rather than dynamically grouping user interactions based on periods of inactivity.DefaultTrigger fires exactly once when the watermark passes the end of the window, offering no early or speculative firings.discardingFiredPanes()) causes late firings to emit only delta increments rather than the required updated aggregate totals.The goal was to compute non-overlapping 15-minute tumbling aggregations without intermediate updates or local reference data.
This approach places all elements into a single unbounded GlobalWindow, firing whenever a fixed batch count of 100 elements is observed, while querying BigQuery directly via JDBC.
GlobalWindows does not segment stream events into distinct temporal user sessions or inactivity gaps.AfterCount) fire based entirely on batch sizes rather than event-time progression or inactivity periods, ignoring allowed lateness semantics.You required key-agnostic micro-batching into an external database based solely on element volumes.
Answer them yourself and get your score the moment you finish. No account needed.
Full access includes
Keep the momentum going with these hand-picked practice scenarios
Last updated Validated