Notesstream🌱 seedling
Log-Based Message Broker
Also: Log-Based Broker · Event Log · Distributed Log · Commit Log · Event Streaming Platform · Topic Partition
A broker that stores messages as an append-only, ordered, partitioned log and keeps them until retention removes them, whether or not anyone has read them. Consumers are just positions in the log, so reading is cheap, fan-out is free and history can be replayed.
Why it exists
Queue brokers forget a message once it's acked: no replay, no second reader, no ordering guarantees after redelivery. Databases already had a durable ordered log inside them (the Write-Ahead Log, the Replication Log). A log broker makes that structure the product.
How it works
- A topic is split into partitions. Each partition is an append-only log, replicated with Leader-Based Replication, and every record gets the next offset.
- The producer chooses the partition, usually by Hash Partitioning the record key, so all records for a key share a partition and stay in order.
- Reading changes nothing on the broker. Each reader stores a Consumer Offset.
- Parallelism comes from partitions: a Consumer Group assigns each partition to one member.
- Records leave by retention (Kafka default 7 days), not by consumption, or by Log Compaction for keyed state.
Trade-offs
| Gains | Costs |
|---|---|
| replay from any retained offset; new readers start at 0 | order only per partition, never across |
| many consumer groups at no extra copy | parallelism capped by partition count |
| high throughput (sequential I/O, no per-message state) | a slow or poison record blocks its partition |
In practice
- Kafka at LinkedIn (2019): 100+ clusters, 4,000+ brokers, 7 million partitions, 7 trillion messages a day.
- Pulsar separates stateless brokers from BookKeeper storage, and its shared subscriptions add queue semantics on top.
Pitfalls
- Assuming global ordering across partitions.
- Adding partitions changes hash(key) mod n, so a key's new records move to a different partition from its old ones.
- A hot key creates a hot partition that one consumer must handle alone.
Connections
- Part of
- Message Broker
- Builds on this
- Kappa ArchitectureStateful Stream Processing
- Related
- Change Data CaptureEvent SourcingHash PartitioningReplication LogTotal Order BroadcastWrite-Ahead Log
- In real systems
- Apache KafkaRabbitMQ
- Mentioned
- Leader-Based Replication
Recall
These cards are also in your review deck, scheduled with spaced repetition.