Dataflow Streaming Windowing for Late Data

Windowing strategies for temporal aggregation divide an unbounded streaming dataset into finite time intervals so each interval can be aggregated separately. In Dataflow (opens in a new tab), which runs Apache Beam (opens in a new tab) pipelines, windowing is applied with Window.into() before a per-key aggregation. The source is typically Pub/Sub (opens in a new tab) delivering event messages to the pipeline. The choice of window type and duration controls when results are emitted, how complete the results are, and how much compute and sink throughput the pipeline consumes.

1.1Fixed Windows

Fixed windows assign streaming events to equal-size, non-overlapping time intervals based on event time. The intervals are defined by a window duration, such as every 30 seconds, using Window.into(FixedWindows.of(Duration.standardSeconds(30))) or FixedWindows.of(DurationUtils.parseDuration(options.getWindowDuration())). Each key produces one aggregate per window, so a pipeline that extracts user_id and click_count pairs and then applies Sum.longsPerKey() emits one total per user for each interval. Choose fixed windows when the business question needs regular reports, such as counts per minute, totals per hour, or windowed files per interval.

1.2Sliding and Session Windows

Sliding windows and session windows are alternative strategies for grouping the same Pub/Sub event stream by event time when fixed intervals do not match the question. Sliding windows use overlapping intervals defined by a size and a period, so one event can belong to more than one window, which is useful for moving averages and rolling totals. Session windows group events by periods of activity separated by a gap of inactivity, so window boundaries are determined by the data rather than the clock, which is useful for grouping user sessions or device bursts. The trade-off between the three strategies is resource consumption:

• Fixed windows keep one copy of each event in one window, so state and downstream writes stay lowest.
• Sliding windows duplicate events across overlapping windows, so state, compute, and sink writes grow as overlap increases.
• Session windows create a variable number of windows that merge and split as events arrive, so state and timing are less predictable under bursty traffic.

1.3Configuring Windows and Windowed Output in Dataflow

Configuring windows in Dataflow means applying Window.into() to a PCollection after reading from the streaming source and before aggregation and writing to the sink. The change-data-capture flow shows the full temporal aggregation path from source to sink:

1. Read: The pipeline reads change events from the streaming source into a PCollection<Row>.
2. Apply window: Data flows from the read step into Window.into(FixedWindows.of(...)), which assigns each row to its time interval.
3. Extract key and value: Data flows from the windowed collection through a mapping step that produces key-value pairs, such as user ID to click count.
4. Aggregate: Data flows from the mapping step into Sum.longsPerKey(), which combines values for each key within each window.
5. Format and write: Data flows from the aggregation step into formatting and then into the sink, such as another table or Cloud Storage.

Window duration is often exposed as a pipeline option, so the same code can run with different intervals without rebuilding. For file sinks, the pipeline also enables windowed writes with TextIO.write().withWindowedWrites() and a WindowedFilenamePolicy that builds the output directory, filename prefix, shard template, suffix, and year, month, day, hour, and minute patterns from each window. That policy lets each window write its own set of files, so downstream consumers can process one interval at a time.

1.4Balancing Latency, Completeness, and Resource Use

The window strategy in a Dataflow pipeline fed by Pub/Sub balances latency, completeness, and resource consumption through window size, overlap, and sink behavior. Shorter fixed windows emit results sooner and lower end-to-end latency, but they create more windows, more aggregation outputs, and more writes to sinks such as BigQuery or Cloud Storage. Longer windows wait for more events and produce fewer, more complete intervals, but results arrive later and workers must hold window state longer. Sliding windows improve smoothness of moving metrics at the cost of repeated computation, while session windows improve grouping of irregular activity at the cost of variable state.

Resource pressure also comes from outside the windowing code, and Dataflow runs pipeline stages in parallel across multiple workers so external limits become the upper bound. Three outside factors shape whether a chosen window configuration can scale:

• Scalability of external systems: A source, sink, or enrichment service can constrain throughput, such as a Kafka topic with too few partitions or a quota-limited API.
• Choice of data formats: Formats that support parallel reads, such as Avro, are usually faster than CSV with embedded newlines or compressed files that cannot be split easily.
• Data location and network topology: Geographic distance and networking between the Dataflow workers in their region and the Pub/Sub topic, storage bucket, or database affect performance.

Because Dataflow parallelizes work and can autoscale workers, calling external services per element can overwhelm quotas, and adding workers to a slow external call can add further pressure. Batching elements into fewer requests, using managed sources and sinks with built-in scalability, and matching window duration and shard count to Pub/Sub throughput quotas keep latency low without exhausting compute, quota, or sink capacity.

Streaming pipeline state management and correctness is how windowed state is built, when partial results are emitted, and how sinks handle repeats caused by retries and late data. In Cloud Dataflow (opens in a new tab), an Apache Beam pipeline reads from a durable source such as Pub/Sub, groups events into windows, maintains per-key state with a combine, and writes to a managed sink such as BigQuery (opens in a new tab), Bigtable, or Apache Iceberg. Correctness depends on controlling state size, emission timing, and sink behavior together.

2.1Windowed Aggregation in Streaming Pipelines

Windowed aggregation assigns each event to a time window and then computes a per-key result for that window. The documentation example reads change data capture events from an Iceberg table as rows with user_id and click_count, applies Window.into(FixedWindows.of(Duration.standardSeconds(30))), maps each row to a key-value pair of user_id to click_count, applies Sum.longsPerKey(), and formats the result back to a row with user_id and total_clicks. Data flows from the managed change-data-capture read to the window assignment, then to the key extraction, then to the per-key sum, then to the managed Iceberg write. Because the sum is computed per key per 30-second window, late or out-of-order events assigned to the same window update the same windowed state rather than creating unbounded global state.

2.2Stateful Processing with GroupByKey and Combine

Stateful processing in Beam holds intermediate values by key and window while a combine such as Sum, Combine.PerKey, Combine.Globally, or Count.PerKey merges them. The pipeline depends on the key distribution, because Dataflow runs stages in parallel across workers and all values for one key must be processed together to produce a correct total. A single hot key with very high volume creates a bottleneck where one worker holds disproportionate state and throughput stalls, which appears as growing backlog_bytes and backlog_elements for the fused combine step. Correctness therefore requires both the right combine and a strategy for key skew when volume is uneven.

Mitigation options in the documentation preserve the same logical result while spreading state and work:

• Rekey data with a ParDo transform to output new key-value pairs that distribute load before the combine.
• Use .withFanout on combine transforms to add intermediate partial combines.
• Use Combine.Globally.withFanout instead of plain Combine.Globally for high-volume unbounded PCollections in Java.
• Use Combine.PerKey.withHotKeyFanout instead of Count.PerKey when one key dominates a per-key count.

2.3Emitting Timely Results with Polling and Triggering Frequency

Timely emission controls how often a streaming pipeline observes new input and flushes incremental updates to the sink. The Iceberg change-data-capture read uses streaming set to true with poll_interval_seconds set to 20, so the source checks the table for new append-only snapshots every 20 seconds and emits newly discovered events downstream. The Iceberg streaming write uses triggering_frequency_seconds set to 20, so buffered rows are flushed to the destination table on that interval rather than waiting for a window to close. The test generator pattern shows the same idea with GenerateSequence.from(0).withRate(1, Duration.standardSeconds(5)), which produces one element every 5 seconds to drive continuous windowed flow. Together, polling on the read side and triggering frequency on the write side bound latency from source snapshot to sink-visible update, which matters when downstream windows aggregate user clicks every 30 seconds.

2.4Idempotent Sinks and Exactly-Once Correctness

An idempotent sink produces the same final table state even if the same result is written more than once after a retry, replay, or late-data update. Dataflow pipelines favor Google Cloud managed sources and sinks because they provide built-in scalability, durable buffering, and quota headroom that reduce the need for custom retry logic. Pub/Sub acts as a scalable and durable event ingestion service that delivers messages to and from streaming pipelines, so an upstream replay can be reprocessed without losing events. On the write path, native Beam I/O transforms batch multiple elements into single requests, with BigQueryIO given as the example for automatic batching of streaming inserts, and Cloud Storage writes use temporary files that are finalized only when the write completes.

Exactly-once effects require controlling both the sink and the calls around it. External service calls add per-call overhead and quota risk, and autoscaling can worsen the problem by adding workers to a slow external-call step, which adds further pressure on the external system. The pipeline therefore should batch external calls, respect daily and per-second quotas, and ensure sufficient Compute Engine quota so autoscaling does not stall and cause repeated work. Failed or unparsable rows should be isolated through a dead-letter pattern, where the example builds a DeadLetterQueueManager rooted at a dlq/ directory under the temporary location, so bad records do not block correct windows or force full replays. Monitoring backlog_bytes, Pub/Sub quota usage, and BigQuery Storage API concurrent connection count reveals whether retries, throttling with 429 (Rate Limit Exceeded), or a stuck combine is threatening correctness before duplicates reach the sink.

Handling late and out-of-order data is the set of controls that lets a streaming pipeline produce correct event-time results even when events arrive after earlier events or after their window has closed. In Apache Beam (opens in a new tab) running on Cloud Dataflow (opens in a new tab), three mechanisms work together: watermarks track event-time progress, allowed lateness keeps window state open for corrections, and discard or side-output policies decide the fate of data that arrives too late. Together they balance result accuracy against state size, latency, and cost.

3.1Watermark Generation and Event-Time Tracking

Watermark generation is the technique a source uses to estimate that event time is complete up to a given timestamp. The watermark flows from the source to downstream windowed transforms, and a window whose end time is below the watermark is considered ready for its on-time result. Sources derive watermarks from event timestamps carried in systems such as Cloud Pub/Sub, from custom timestamp policies in file or Kafka sources, and from heuristic progress when perfect ordering is impossible. A heuristic watermark advances based on observed event times and arrival patterns, so out-of-order events below the watermark are classified as late even if they belong to an earlier window. Idle partitions, stalled shards, or a source without new events can hold the watermark back, which delays on-time firings for all keys downstream until progress resumes.

3.2Allowed Lateness Thresholds and Window Lifecycle

Allowed lateness is a per-window threshold that keeps window state alive after the watermark has passed the end of the window. The threshold extends the window lifecycle so late events can still update the result instead of being dropped immediately at watermark time. The lifecycle proceeds in order:

1. Assigned and open: Each event is assigned to its event-time window and state accumulates while the watermark is below the window end.
2. Watermark passes window end: The pipeline emits the on-time pane and the window becomes closing, but state is retained for the allowed-lateness duration.
3. Late arrivals within allowed lateness: Late events flow into the retained state and trigger refinements or late panes, depending on the trigger configuration.
4. Garbage collection: Only after the window end plus allowed lateness has passed the watermark does the runner discard the window state, and any later event is excessively late.

A longer threshold improves completeness for delayed mobile, IoT, or retry traffic but retains state longer and increases persistent state and shuffle cost. A short or zero threshold frees state quickly and emits final results sooner, at the price of dropping more late data.

3.3Discarding, Updating, and Side-Output Policies for Excessively Late Data

Excessively late data is data that arrives after garbage collection, when its window state no longer exists on the runner. The excessively late data policy must be explicit because the main output can no longer be correctly updated without unbounded state. Accumulation mode controls how refinements before that point behave, while the late-data policy controls what happens after that point.

The side-output collection depends on the main windowed transform but has its own destination and schema, so late events can flow from the streaming pipeline to a batch repair job without blocking on-time results. Use discarding when late arrivals are rare noise, and use side outputs when late arrivals represent revenue, safety, or compliance events that require a separate correction path.

3.4Controlling State Size While Preserving Accuracy

Controlling state size while preserving accuracy requires tuning allowed lateness, trigger frequency, and accumulation together. Frequent refinements with accumulating mode improve freshness but hold larger buffers and emit more panes to the sink, which increases write and downstream merge cost. Discarding mode emits smaller, independent panes and uses less state, but downstream systems must replace rather than combine results. For high-volume keyed streams, prefer short allowed lateness with side outputs for outliers, so the common path stays fast and cheap while the rare late path is preserved for audit and backfill without retaining every window indefinitely.

Key Facts

• Fixed windows assign each event to one non-overlapping interval, while sliding windows duplicate events across overlapping intervals and session windows group events by activity gaps.
• Dataflow applies Window.into() after reading from Pub/Sub and before per-key aggregation such as Sum.longsPerKey(), with windowed file sinks using withWindowedWrites() and a window-based filename policy.
• A hot key forces one worker to hold disproportionate state for a combine, which appears as growing backlog_bytes and backlog_elements and is mitigated with rekeying, withFanout, or withHotKeyFanout.
• The watermark tracks event-time progress and decides what is late, while allowed lateness keeps window state alive after the watermark passes so late events can still update results before garbage collection.
• Excessively late events that arrive after state is discarded must be dropped and counted or routed to a side output for audit and reprocessing.