Notesreplication🌱 seedling
Failover
Also: Automatic Failover · Leader Promotion · Promotion
Promoting a follower to leader when the leader fails, and pointing clients and other followers at it. Every step (detecting, choosing, reconfiguring) has a failure mode.
Why it exists
A dead follower just restarts and catches up. A dead leader means no writes at all until someone takes over.
How it works
- Detect the failure, usually by a Timeout: crashed, slow and partitioned look identical from outside.
- Choose a new leader, ideally the most up-to-date follower, by Leader Election or a controller.
- Reconfigure clients to send writes to it, and make sure the old leader steps down when it returns.
Trade-offs
| Timeout | Effect |
|---|---|
| short | fast recovery, but load spikes and network blips cause needless failovers that add load |
| long | fewer false alarms, but longer write outages |
In practice
- Patroni (PostgreSQL) holds the leader role as a key with a TTL in etcd, Consul or ZooKeeper (30 s by default) and limits promotion to followers within
maximum_lag_on_failover. - GitHub, September 2012: a node already known to be out of date was promoted. MySQL-generated IDs were also keys in Redis, so the stores disagreed: events showed on the wrong dashboards and 16 private repositories were briefly exposed.
Pitfalls
- With asynchronous replication the new leader lacks the old leader's last writes; they're usually discarded, which is dangerous if caches, indexes or other databases already saw them.
- The old leader returning and still acting as leader: Split Brain.
- An untested failover is a failover that doesn't work.
Connections
- Part of
- Leader-Based Replication
- Learn first
- Failure Detection
- Related
- Split BrainLeader ElectionTimeoutSynchronous vs Asynchronous ReplicationSingle Point of Failure
- In real systems
- MySQLPostgreSQL
Recall
These cards are also in your review deck, scheduled with spaced repetition.