Notesstream🌱 seedling
Watermark
Also: Low Watermark · Event-Time Watermark · Late Data · Allowed Lateness · Grace Period · Idle Source · Trigger
A watermark is a timestamp flowing through a streaming job that asserts "no more events older than this are expected". Windows fire when it passes their end. It's a heuristic: too eager produces late data, too cautious adds latency.
Also in: Stream Processing › Late data
Why it exists
In event time you can never be sure a window is complete. A watermark turns that uncertainty into an explicit, tunable decision about when to emit results and free their state.
How it works
- Bounded out-of-orderness: watermark = max event time seen − bound (Flink subtracts one more millisecond).
- An operator with several inputs takes the minimum of their watermarks. One idle input pins it, and withIdleness excludes inputs that have gone silent.
- Watermark alignment (Flink 1.15) pauses sources that race too far ahead of the others.
- Late data (behind the watermark) is dropped by default, can go to a side output, or, with allowed lateness, re-fires the window with a corrected result.
- Triggers decide when to emit (early, on time, late). Accumulation modes say whether each emission replaces, adds to or retracts the previous one.
Trade-offs
| Setting | Gains | Costs |
|---|---|---|
| small bound | fresh results | more late data |
| large bound | completeness | every result delayed by the bound |
| allowed lateness | corrected results | state held longer; downstream must accept updates |
In practice
- Flink: WatermarkStrategy.forBoundedOutOfOrderness(...).withIdleness(...). sideOutputLateData catches what's dropped.
- Kafka Streams calls allowed lateness the grace period. Since 3.0 there is no implicit 24-hour default.
Pitfalls
- An idle Kafka partition silently stops all windows from firing: no errors, no output.
- Marking a source idle when its producer is actually stuck turns its events into late data.
- Appending every re-firing downstream double-counts. Upsert by window instead.
Connections
- Part of
- Stream Processing
- Builds on this
- Stream Join
- In real systems
- Apache Flink
Recall
These cards are also in your review deck, scheduled with spaced repetition.