Notesstream🌱 seedling
Consumer Offset
Also: Offset · Committed Offset · Consumer Position · Offset Commit · Consumer Lag
A log consumer's entire progress is one number per partition: the offset of the next record to read. The committed offset is the saved copy a restart resumes from, and where you commit it relative to your side effects decides the delivery guarantee.
Why it exists
A log broker keeps no per-message delivery state, so consumers must remember how far they've read. The offset makes that memory tiny and makes rewinding (replay) trivial.
How it works
- Position: the next offset the consumer will fetch. It advances on every poll.
- Committed offset: the position saved durably. After a crash or a rebalance, the new owner of the partition starts here.
- Kafka stores commits in the compacted internal topic __consumer_offsets (50 partitions by default).
- Auto-commit (on by default, every 5 s) commits what the previous poll returned, not what your code has finished.
- Lag = log end offset − committed offset. It's the main health metric for a consumer.
- Offsets of a group with no members expire after offsets.retention.minutes (7 days). Then auto.offset.reset (default latest) picks the restart point.
Trade-offs
| Commit timing | Result |
|---|---|
| before processing | at-most-once: a crash skips records |
| after processing | at-least-once: a crash replays records |
| atomically with the output | exactly-once (Kafka transactions, or offsets stored with the result) |
In practice
- Kafka: commitSync / commitAsync, or sendOffsetsToTransaction to commit offsets inside a producer transaction.
- Storing offsets in the same database transaction as the results is a classic way to get exactly-once into an external store.
Pitfalls
- Auto-commit combined with handing records to a thread pool loses records on a crash.
- auto.offset.reset=latest silently skips a backlog when offsets expire; use earliest where completeness matters.
- Health checks that restart consumers whose offset stops moving can cause rebalance loops.
Connections
- Part of
- Log-Based Message Broker
- Builds on this
- Consumer GroupStateful Stream Processing
- In real systems
- Apache Kafka
Recall
These cards are also in your review deck, scheduled with spaced repetition.