Notesstream🌱 seedling
Message Broker
Also: Message Queue · Message Bus · Pub/Sub · Publish-Subscribe · Event Broker · Competing Consumers
A server that durably stores messages between producers and consumers, so neither has to be up, fast or even known to the other. It trades a synchronous answer for decoupled availability, buffering and fan-out.
Why it exists
A direct call couples the caller to the callee's uptime, speed and existence. If email is down, orders fail; every new consumer means a producer change. A broker turns "must be up now" into "must eventually catch up".
How it works
- The producer sends a message and gets an ack once the broker has stored it durably; delivery happens later, asynchronously.
- Queue brokers (RabbitMQ, SQS) track every message's state. Each message goes to one of several competing consumers, and it's deleted once acked.
- Log brokers keep messages in order after they are read, and consumers track their own position. See Log-Based Message Broker.
- Unacked messages are redelivered after a disconnect or a visibility timeout, so delivery is at-least-once by default. See Delivery Semantics.
- The broker absorbs bursts: it is a buffer, and consumers pull at the rate they can handle, a form of Backpressure.
Trade-offs
| Gains | Costs |
|---|---|
| producer independent of consumer uptime and speed | no synchronous result; eventual effects |
| add consumers without touching the producer | another stateful system to run |
| absorbs load spikes | duplicates and reordering on redelivery |
In practice
- RabbitMQ: pushes to consumers up to a prefetch limit of unacked messages; quorum queues replicate with Raft.
- SQS: a received message is hidden for a visibility timeout (30 s default, 12 h max) and retained 4 days by default (14 max).
Pitfalls
- Using a broker for request/response ("was the card approved?") just rebuilds RPC with worse latency.
- An unbounded backlog hides an outage: alert on queue depth and age of the oldest message.
Connections
- Part of
- Stream Processing
- Contrasts with
- RPC
- Related
- BackpressureMessage Passing
- In real systems
- RabbitMQApache Kafka
Recall
These cards are also in your review deck, scheduled with spaced repetition.