Notesdistributed🌱 seedling
Coordination Service
Also: Lock Service · Distributed Lock Service · Configuration Service · Distributed Lock
A small, consensus-replicated store (usually 3 or 5 nodes) holding the little data a fleet must agree on: who leads, current configuration, membership, locks. Applications use it instead of implementing consensus themselves.
Why it exists
Few teams should implement Paxos or Raft. A shared service gives every application linearizable writes, leases, watches and ordering through a simple API, at the cost of a dependency everyone shares.
How it works
- Data model: a tree of small nodes (ZooKeeper znodes) or a flat key space with revisions (etcd).
- Leases / sessions: keys that disappear when their owner stops renewing (ephemeral znodes, etcd lease-attached keys).
- Watches: clients are notified when a key changes, instead of polling.
- Ordering: sequential znodes and global revisions give every change a monotonically increasing number, usable as a Fencing Token.
- Recipes: Leader Election (create a key, watchers queue behind it), locks, group membership, Service Discovery, config distribution.
Trade-offs
| Gains | Costs |
|---|---|
| correct agreement without writing consensus | a critical shared dependency |
| watches instead of polling | small data and low write rates only |
| automatic release via leases | locks still need fencing at the resource |
In practice
- Chubby (Google, 2006): five replicas per cell; one master served 90,000 clients; designed for locks, most used for naming.
- Kubernetes stores all cluster state in etcd.
- Kafka used ZooKeeper for metadata until KRaft; Kafka 4.0 (March 2025) removed ZooKeeper mode.
Pitfalls
- Treating its locks as mutual exclusion. Jepsen (2020) showed two holders of an etcd lock at once; the protected resource must reject stale tokens.
- Using it as a general database: large values and high write rates hurt the whole fleet.
- Over-trusting its uptime: Google deliberately took the global Chubby down in quarters when real failures hadn't brought it down to its SLO, so services couldn't come to depend on it never failing.
Connections
- Part of
- Consensus
Recall
These cards are also in your review deck, scheduled with spaced repetition.