Notesstream🌱 seedling
Stateful Stream Processing
Also: Keyed State · Checkpointing · Barrier Snapshot · Asynchronous Barrier Snapshotting · Savepoint · State Backend · Changelog Topic
Stream operators keep local, key-partitioned state (counts, windows, join buffers) and make it fault-tolerant by periodically snapshotting it together with the input offsets it corresponds to. Recovery restores both and replays from there, so each record affects state exactly once.
Also in: Stream Processing › Checkpoints
Why it exists
Aggregates, windows and joins need memory of the past. Querying a remote database per event is slow and races with itself. Local state is fast, but a crash deletes it, so the snapshot must be consistent with the input position.
How it works
- State is partitioned by key alongside the input, so each parallel task owns its keys' state.
- Barrier snapshots (Flink, a Chandy–Lamport variant): sources record their offsets and inject a barrier. Operators with several inputs align barriers, snapshot their state asynchronously, and forward the barrier. Once every task has acked, the checkpoint is complete.
- Recovery: load the last complete checkpoint and rewind sources to its offsets. Replayed records rebuild identical state.
- Unaligned checkpoints let barriers overtake buffered records (stored with the snapshot), so checkpoints still finish under Backpressure.
- Kafka Streams instead backs each local RocksDB store with a compacted changelog topic and restores by replaying it.
Trade-offs
| Choice | Gains | Costs |
|---|---|---|
| heap state | fastest | limited by memory and GC |
| RocksDB / LSM-Tree state | far larger than memory; incremental checkpoints | serialization per access |
| disaggregated (remote) state | fast rescaling and recovery | remote I/O; newer, less proven |
| short checkpoint interval | less replay after a crash | more overhead |
In practice
- Flink: HashMapStateBackend is the default, EmbeddedRocksDBStateBackend supports incremental checkpoints, and ForSt (2.0) puts state on object storage. Savepoints are user-owned snapshots for upgrades and rescaling.
Pitfalls
- Exactly-once state isn't exactly-once output: sinks may have written before the crash.
- Unbounded state (regular joins, global windows, no TTL) eventually makes checkpoints time out.
Connections
- Part of
- Stream Processing
- Learn first
- Log-Based Message BrokerConsumer Offset
- Builds on this
- Exactly-Once SemanticsStream Join
- In real systems
- Apache FlinkRocksDB
Recall
These cards are also in your review deck, scheduled with spaced repetition.