Notesstream🌱 seedling

Stateful Stream Processing

Also: Keyed State · Checkpointing · Barrier Snapshot · Asynchronous Barrier Snapshotting · Savepoint · State Backend · Changelog Topic

Stream operators keep local, key-partitioned state (counts, windows, join buffers) and make it fault-tolerant by periodically snapshotting it together with the input offsets it corresponds to. Recovery restores both and replays from there, so each record affects state exactly once.

Also in: Stream Processing › Checkpoints

Why it exists

Aggregates, windows and joins need memory of the past. Querying a remote database per event is slow and races with itself. Local state is fast, but a crash deletes it, so the snapshot must be consistent with the input position.

How it works

  • State is partitioned by key alongside the input, so each parallel task owns its keys' state.
  • Barrier snapshots (Flink, a Chandy–Lamport variant): sources record their offsets and inject a barrier. Operators with several inputs align barriers, snapshot their state asynchronously, and forward the barrier. Once every task has acked, the checkpoint is complete.
  • Recovery: load the last complete checkpoint and rewind sources to its offsets. Replayed records rebuild identical state.
  • Unaligned checkpoints let barriers overtake buffered records (stored with the snapshot), so checkpoints still finish under Backpressure.
  • Kafka Streams instead backs each local RocksDB store with a compacted changelog topic and restores by replaying it.

Trade-offs

Choice Gains Costs
heap state fastest limited by memory and GC
RocksDB / LSM-Tree state far larger than memory; incremental checkpoints serialization per access
disaggregated (remote) state fast rescaling and recovery remote I/O; newer, less proven
short checkpoint interval less replay after a crash more overhead

In practice

  • Flink: HashMapStateBackend is the default, EmbeddedRocksDBStateBackend supports incremental checkpoints, and ForSt (2.0) puts state on object storage. Savepoints are user-owned snapshots for upgrades and rescaling.

Pitfalls

  • Exactly-once state isn't exactly-once output: sinks may have written before the crash.
  • Unbounded state (regular joins, global windows, no TTL) eventually makes checkpoints time out.

Recall

These cards are also in your review deck, scheduled with spaced repetition.