Notesfoundations🌱 seedling
Software Fault
Also: Software Error · Systematic Fault · Correlated Failure
Software faults are systematic: the same bug is in every copy of the code, so one trigger hits every replica at once. Correlated faults defeat redundancy, which makes them more dangerous than hardware faults.
Why it exists
Bugs hide behind assumptions about the environment that are true almost always, until they aren't.
How it works
Common shapes:
- A bug triggered by a particular input or event. The 2012 leap second (23:59:60) hit a Linux kernel bug, and many applications hung or spun at 100% CPU at the same moment.
- A runaway process exhausting a shared resource: CPU, memory, disk or bandwidth.
- A dependency that slows down, stops responding, or returns corrupted data.
- A Cascading Failure: one fault triggers the next. A slow dependency ties up callers' threads until they stop answering too.
Trade-offs
| Mitigation | Cost |
|---|---|
| thorough testing, process isolation, crash-and-restart | engineering time |
| Timeouts, Circuit Breakers, Load Shedding | tuning; may reject requests that would have succeeded |
| self-checks (e.g. a queue verifies messages in = messages out) | instrumentation |
In practice
- Production monitoring and measurement catch what tests miss. Systems that can check their own guarantees can alarm on a discrepancy.
- In a study of Cassandra, HBase, HDFS, MapReduce and Redis, 92% of catastrophic failures came from mishandling errors the code had already detected. Error-handling paths deserve their own tests.
Pitfalls
- "We have replicas" doesn't help: every replica shares the bug.
- Slow is worse than down. Dead fails fast; slow holds callers hostage. Timeouts turn slow into fast failure.
Connections
- Part of
- Reliability
- Learn first
- Fault Tolerance
- Mentioned
- TimeoutCircuit BreakerLoad Shedding
Recall
These cards are also in your review deck, scheduled with spaced repetition.