Notesstream🌱 seedling
Stream Join
Also: Streaming Join · Interval Join · Windowed Join · Stream-Table Join · Temporal Join · Lookup Join · Stream Enrichment
A streaming join must remember one side until the matching record from the other side arrives, so its cost is state. Time bounds (an interval, a window, a versioned table) tell it when it may forget; without them, state grows forever.
Why it exists
Enrichment (orders + customer data), attribution (clicks + impressions) and correlation (payments + fraud signals) all need records from different streams combined, while neither stream waits for the other.
How it works
- Stream–stream (interval / windowed): buffer both sides in keyed state, emit matches within the interval, and evict once the Watermark passes the interval's end.
- Stream–table (temporal): keep the table as state, built from a changelog (often a compacted or CDC topic). Each stream record looks up the version valid at its event time, which requires a versioned table.
- Lookup: query an external store per record at processing time. Little state, but non-deterministic on replay.
- Table–table (regular): maintain a joined Materialized View of two changelogs, holding both sides forever unless bounded by a TTL.
Trade-offs
| Join | Gains | Costs |
|---|---|---|
| interval | bounded state, deterministic | misses matches outside the bound |
| temporal | correct as-of-event-time enrichment | versioned table; watermarks on both sides |
| lookup | tiny state, always fresh | external load; different answers on replay |
| regular | full relational semantics | unbounded state unless TTL (which drops matches) |
In practice
- Kafka Streams: KStream–KStream joins are always windowed. KStream–KTable joins trigger only on stream records.
- Flink SQL: regular joins keep both inputs in state forever, and table.exec.state.ttl bounds them at a correctness cost. One published case replaced regular joins with temporal joins and cut state from about 780 GB to checkpoints under 4 GB.
Pitfalls
- Regular joins on unbounded streams grow state until checkpoints time out.
- A stream–table join that is non-deterministic on replay, because the table has changed since.
- Forgetting that both inputs need to be co-partitioned by the join key.
Connections
- Part of
- Stream Processing
- Learn first
- WindowingWatermarkStateful Stream Processing
- In real systems
- Apache Flink
Recall
These cards are also in your review deck, scheduled with spaced repetition.