Notesstream🌱 seedling
Consumer Group
Also: Consumer Group Rebalance · Cooperative Rebalancing · Static Membership · Share Group · KIP-848
Consumers sharing a group ID split a topic's partitions so that each partition has exactly one owner in the group. A rebalance reassigns partitions as members join, leave or die, and how it's done decides whether the whole group pauses.
Why it exists
One consumer can't keep up with a busy topic, but per-key order requires one reader per partition. A group gives parallelism across partitions and exclusivity within each. Separate groups each receive every record.
How it works
- A broker-side group coordinator tracks members through heartbeats and assigns partitions.
- Eager rebalance (original): every member revokes all partitions, rejoins, and waits for the new assignment. The whole group stops.
- Cooperative (Kafka 2.4, KIP-429): members keep unchanged partitions and give up only those that move.
- Static membership (Kafka 2.3, KIP-345): a stable group.instance.id lets a restarted member reclaim its partitions with no rebalance.
- KIP-848 (GA in Kafka 4.0, opt-in via group.protocol=consumer): the coordinator computes assignments and pushes changes through heartbeats, with no group-wide barrier.
- Members are removed after session.timeout.ms (45 s) without heartbeats, or after max.poll.interval.ms (5 min) without polling.
Trade-offs
| Gains | Costs |
|---|---|
| horizontal scaling with per-partition order | parallelism ≤ partition count; extras idle |
| automatic failover of partitions | rebalances pause work; duplicates replay from last commit |
| independent groups, each at its own offset | tuning timeouts trades detection speed for false evictions |
In practice
- Share groups (KIP-932, production-ready in Kafka 4.2) add queue semantics: per-record acks, delivery-count limits, and more consumers than partitions, at the cost of ordering.
- Uber built a consumer proxy with per-message acks and DLQs after hitting rebalance storms and the partition cap.
Pitfalls
- A batch that takes longer than max.poll.interval.ms gets the member evicted, and the batch is redone elsewhere, possibly in a loop.
- Rolling restarts of large groups under the eager protocol cause back-to-back rebalances.
- Restarting consumers because their offset stopped moving can itself trigger endless rebalances.
Connections
- Part of
- Log-Based Message Broker
- Learn first
- Consumer Offset
- In real systems
- Apache Kafka
Recall
These cards are also in your review deck, scheduled with spaced repetition.