Notesreplication🌱 seedling
Split Brain
Also: Split-Brain · Dual Leader · Dual Primary
Two nodes both acting as leader and accepting writes, so their histories diverge and can't be merged automatically. It happens when an old leader is cut off rather than dead.
Why it exists
Failover assumes the old leader is gone. A Network Partition makes a healthy leader look dead to the rest of the cluster, which promotes a replacement while the old one keeps serving the clients that can still reach it.
How it works
- Partition → the majority side promotes a new leader → both sides accept writes → each holds writes the other lacks.
- When the partition heals, choosing either history discards acknowledged writes from the other.
- Prevention is about proving leadership before writing:
- a Lease granted by a majority, which the leader must renew or stop writing;
- a Fencing Token (an epoch number) that storage checks, rejecting writes from older leaders;
- STONITH: power off the old leader before promoting.
Trade-offs
| Defence | Cost |
|---|---|
| majority lease | needs 3+ voters (or a coordination service); minority side becomes read-only |
| fencing tokens | every storage layer must check them |
| STONITH | needs out-of-band power control; can kill a node that was fine |
In practice
- GitHub, October 2018: a 43-second partition between its East Coast hub and primary data centre led Orchestrator to promote West Coast databases. Both sides then held unreplicated writes (954 in the window on one busy cluster), and the site was degraded for 24 hours 11 minutes while GitHub restored and reconciled.
Pitfalls
- Fencing stops new writes on the old leader, but not ones it accepted before noticing. With asynchronous cross-region replication some loss on failover remains.
- Two-node clusters can't form a majority on either side of a partition; add a witness.
Connections
Recall
These cards are also in your review deck, scheduled with spaced repetition.