Notesstream🌱 seedling
Kappa Architecture
Also: Kappa · Reprocessing by Replay · Stream Reprocessing
One stream processing codebase over a log retained long enough to replay. To change the logic, run a second version of the job from the start of the log into a new table, switch readers when it catches up, then delete the old one.
Why it exists
The Lambda architecture duplicates logic across batch and streaming. If the log keeps enough history, batch is just a replay of a bounded stream, so one job can do both.
How it works
- Retain the input log for as long as you may need to reprocess (to reprocess 30 days, keep 30 days, or use tiered storage).
- Start job v2 from offset 0, writing to a new output table, while v1 keeps serving.
- When v2 has caught up to the live end, compare outputs, then switch the application to v2's table.
- Stop v1 and delete its table. Rolling back means switching back while v1 still exists.
Trade-offs
| Gains | Costs |
|---|---|
| one codebase, one semantics | long log retention (storage cost) |
| reprocessing is a routine deploy | replay must be fast: partitions limit parallelism |
| old and new outputs can be compared | jobs must be deterministic over event time |
In practice
- Kafka tiered storage (KIP-405; early access in 3.6, production-ready in 3.9 in 2024) makes months of retention cheap.
- Uber (2020) backfilled nine days, about 10 TB, through the same Spark Streaming job by reading Hive as a streaming source.
- LinkedIn (2023) ran one Beam pipeline as both stream and backfill; the backfill fell from seven hours to 25 minutes.
Pitfalls
- Wall-clock time, randomness or lookups to mutable services make replay give different results.
- Retention shorter than the oldest view you may need to rebuild.
Connections
- Part of
- Stream Processing
- Learn first
- Log-Based Message BrokerDerived Data
- Contrasts with
- Lambda Architecture
- In real systems
- Apache KafkaApache Flink
Recall
These cards are also in your review deck, scheduled with spaced repetition.