Notesfoundations🌱 seedling

Fault Tolerance

Also: Fault-Tolerant · Resilient · Fault vs Failure

A fault is one component deviating from its spec. A failure is the whole system no longer serving users. Fault tolerance is designing so faults don't become failures.

Why it exists

You can't make the probability of a fault zero. So reliability comes from containing faults, not from preventing them all.

How it works

  • Decide which specific faults you'll tolerate (e.g. "any one machine can die"). No system tolerates every fault.
  • Put boundaries where faults must stop: Redundancy, health checks plus a Load Balancer, isolation, Timeouts and Circuit Breakers.
  • Recover fast: automatic failover, restarts, rollbacks.
  • Keep the mechanisms exercised. Tolerance code that never runs rots, which is why Chaos Engineering exists.

Trade-offs

Gains Costs
faults become non-events for users more machines, more moving parts, harder testing

In practice

  • Load balancer + N servers: one server crashes and traffic routes around it.
  • Rolling Upgrades: a system that tolerates losing a node can take nodes down on purpose to patch them.

Pitfalls

  • Using "fault" and "failure" interchangeably. The distinction is the design goal.
  • Claiming "fault-tolerant" without naming which faults.
  • Redundancy against correlated faults (see Software Fault) doesn't tolerate anything.

Recall

These cards are also in your review deck, scheduled with spaced repetition.