Notesdistributed🌱 seedling
Failure Detection
Also: Failure Detector · Phi Accrual Failure Detector · Heartbeat · Accrual Failure Detector
Deciding that a remote node has failed from the absence of messages. Every detector trades detection speed against false suspicions, because over an asynchronous network a slow node and a dead one look identical.
Why it exists
Load balancers, leaders and clusters must stop sending work to dead nodes and trigger Failover. Nobody can observe a crash directly, only silence.
How it works
- Heartbeats + fixed timeout: declare dead after T seconds of silence. Simple; right only for the network it was tuned on.
- Phi accrual: keep a window of recent heartbeat gaps, estimate their distribution, and output φ = −log₁₀(probability a heartbeat would still arrive this late). Callers suspect above a threshold, which adapts as the network gets noisier.
- Stronger signals where available: a switch reporting link down, the OS refusing connections (nothing listening on the port), a process manager reporting an exit. Absence of any signal still needs a timeout.
- A detector says "probably dead". Acting safely on a false suspicion needs Fencing Tokens or a Lease.
Trade-offs
| Timeout | Gains | Costs |
|---|---|---|
| short | fast failover | false positives, duplicate work, failover storms under load |
| long | few false alarms | requests hang on dead nodes; slow recovery |
In practice
- Cassandra: phi accrual with default
phi_convict_threshold8; it models gaps as exponential, so φ = 8 is about 18 mean heartbeat intervals of silence. Docs suggest 10–12 on EC2. - Akka Cluster uses phi accrual; SWIM (Gossip Protocol) adds indirect probes and suspicion.
Pitfalls
- Declaring an overloaded node dead moves its load to others, which then look dead too (Cascading Failure).
- One timeout constant across environments with very different latency.
Connections
- Part of
- Distributed Systems
- Learn first
- TimeoutUnreliable Network
- Builds on this
- FailoverLeader Election
- Related
- Consumer GroupFLP ImpossibilityGossip ProtocolHealth CheckLeaseProcess PauseNetwork Partition
- In real systems
- Cassandra
- Mentioned
- Fencing TokenCascading Failure
Recall
These cards are also in your review deck, scheduled with spaced repetition.