Notesdistributed🌱 seedling

Failure Detection

Also: Failure Detector · Phi Accrual Failure Detector · Heartbeat · Accrual Failure Detector

Deciding that a remote node has failed from the absence of messages. Every detector trades detection speed against false suspicions, because over an asynchronous network a slow node and a dead one look identical.

Why it exists

Load balancers, leaders and clusters must stop sending work to dead nodes and trigger Failover. Nobody can observe a crash directly, only silence.

How it works

  • Heartbeats + fixed timeout: declare dead after T seconds of silence. Simple; right only for the network it was tuned on.
  • Phi accrual: keep a window of recent heartbeat gaps, estimate their distribution, and output φ = −log₁₀(probability a heartbeat would still arrive this late). Callers suspect above a threshold, which adapts as the network gets noisier.
  • Stronger signals where available: a switch reporting link down, the OS refusing connections (nothing listening on the port), a process manager reporting an exit. Absence of any signal still needs a timeout.
  • A detector says "probably dead". Acting safely on a false suspicion needs Fencing Tokens or a Lease.

Trade-offs

Timeout Gains Costs
short fast failover false positives, duplicate work, failover storms under load
long few false alarms requests hang on dead nodes; slow recovery

In practice

  • Cassandra: phi accrual with default phi_convict_threshold 8; it models gaps as exponential, so φ = 8 is about 18 mean heartbeat intervals of silence. Docs suggest 10–12 on EC2.
  • Akka Cluster uses phi accrual; SWIM (Gossip Protocol) adds indirect probes and suspicion.

Pitfalls

  • Declaring an overloaded node dead moves its load to others, which then look dead too (Cascading Failure).
  • One timeout constant across environments with very different latency.

Recall

These cards are also in your review deck, scheduled with spaced repetition.