Reliability
Things will break. A reliable system keeps working anyway. What that means, where faults come from, and how good systems stop a fault from becoming a failure.
The whole idea
What reliable means
Ask people what they expect from software and you get roughly this list. It does what you asked. It copes when you use it in ways nobody planned, including by mistake. It’s fast enough under the load it actually gets. And it doesn’t let the wrong people in.
Reliability means all of that stays true when things go wrong. That’s when, not if: disks die, code has bugs, people mistype commands. A reliable system is one that keeps working correctly anyway.
The things that go wrong are faults. A system that copes with them is fault-tolerant, or resilient. “Fault-tolerant” never means tolerant of every fault. If a black hole swallowed the Earth, you’d need servers in space. It always means tolerant of specific, named kinds of fault.
Faults vs failures
People use these two words as if they meant the same thing. They don’t, and the difference is the whole subject.
- A fault is one component deviating from its spec: a disk returning errors, a process crashing, a network link dropping packets.
- A failure is the system as a whole no longer providing the service its users need.
You can’t make the probability of a fault zero. What you can do is design so that faults don’t turn into failures.
Same fault, two outcomes
A surprising consequence follows: it can make sense to raise the rate of faults on purpose, so the containment walls get tested all the time. We’ll come back to that. The main exception is security. If an attacker has read your data, no failover undoes it, so for security you prevent faults rather than tolerate them.
Hardware faults
One machine rarely breaks. Thousands of machines break all the time. At scale, hardware faults stop being events and become a steady background rate.
Datasheets promise a hard disk a mean time to failure (MTTF) of over a century. Fleets see worse: field studies measure 2–4% of disks failing each year, an MTTF of a few decades, and Backblaze’s fleet of nearly 300,000 drives still ran at 1.57% in 2024. That sounds reassuring until you own ten thousand of them. At 3.65% a year, ten thousand disks lose about 365 a year: one disk every day. RAM, power supplies and network cables add their own steady trickle.
A month in the life of a cluster
healthy disk failed disk · each square is one disk
The good news is that hardware faults are mostly random and independent. One disk dying says little about its neighbour.Mostly. A bad manufacturing batch, a hot rack or a shared power feed can correlate them. That’s what makes the classic defence work: redundancy. RAID keeps copies of data on several disks. Servers have dual power supplies and hot-swappable CPUs. Data centres have batteries and diesel generators. When one part dies, its twin takes over while the broken one is replaced.
Redundancy at the component level
For a long time that was enough: a single well-built machine could run for years. Two things changed. Applications grew to need many machines, which multiplies the fault rate. And cloud platforms favour flexibility over single-machine reliability, so a virtual machine can disappear with no warning at all.
So systems moved to tolerating the loss of entire machines, in software, on top of the hardware redundancy. There’s a lovely side effect: if you can lose any one machine, you can also take one down on purpose. That means you can patch the whole fleet one node at a time with no planned downtime, which is called a rolling upgrade.
Losing a machine, on purpose
Software faults
Hardware faults are random. Software faults are systematic. The same bug lives in every copy of the code, so when its trigger arrives, it strikes everywhere at once. That makes software faults far more dangerous: they are correlated, and correlation defeats redundancy.
When every replica fails together
The leap second is the textbook case: a kernel bug made many applications on Linux hang or spin at the same instant, and sites including Reddit had a very bad night. At Mozilla, Java services (Hadoop, HBase, Elasticsearch) pinned their CPUs at midnight UTC, and the fix, resetting the system clock, went out to every machine through Puppet that night. The other classic shapes:
- A process that runs away with a shared resource: CPU, memory, disk space or network bandwidth.
- A dependency that slows down, stops answering, or starts returning garbage.
- A cascading failure: a small fault in one component triggers a fault in the next, which triggers another.
How a slow dependency takes everything down
There’s no single fix for software faults. They usually hide behind an assumption about the environment that is true almost all the time, until one day it isn’t. What helps is a pile of small disciplines:
- Think carefully about assumptions and about how components interact.
- Test thoroughly.
- Isolate processes, and let them crash and restart cleanly.
- Measure and monitor how the system behaves in production.
A system can also check itself. A message queue should emit exactly as many messages as it took in. If it continually checks that, it can raise an alarm the moment the numbers disagree.
Human error
People design, build and run these systems, and people make mistakes, including careful, well-meaning experts. A 2003 study of three large internet services found that operator error was the leading cause of outages in two of them, and that most of those operator errors were configuration mistakes. Hardware, on servers or in the network, caused between 4% and 25% of the outages.
One typo, two rollouts
Real outages look exactly like this. In February 2017, an input to a routine command was entered incorrectly and
removed far more servers than intended, taking Amazon S3 in us-east-1 down for hours, and a big slice of the
internet with it.In October 2021, a command meant to check backbone capacity instead disconnected all
of Facebook’s data centres. By Cloudflare’s measurements Facebook was unreachable from about 15:40 to 21:28 UTC,
nearly six hours. It was a human action again, and a tooling gap.
You can’t make people perfect, but you can make the system forgiving. The best setups combine all of these:
- Make the right thing easy. Good abstractions, APIs and admin tools steer people away from mistakes. Too restrictive and people route around them, so it’s a balance.
- Give people sandboxes. Separate the places where people make the most mistakes from the places where mistakes cause failures: non-production environments, with real data, where exploring is safe.
- Test at every level, from unit tests to whole-system integration tests to manual checks. Automated tests reach the corner cases that normal operation rarely hits.
- Make recovery fast. Config should roll back in one step. New code should roll out gradually, as canaries, so a bug reaches a few users rather than all of them. Keep tools that can recompute data if a computation turns out to have been wrong.
- Monitor in detail. Performance metrics and error rates (telemetry) give early warning, and they’re the evidence you need when diagnosing.
- Invest in people. Good training and management matter as much as any tool.
Breaking things on purpose
Many serious bugs live in error-handling code: failover paths, retries, recovery. A 2014 study of 198 user-reported failures in Cassandra, HBase, HDFS, Hadoop MapReduce and Redis found that 92% of the catastrophic ones came from mishandling an error the software had already detected. That code rarely runs, so it quietly rots. A failover that has never been exercised probably doesn’t work.
So deliberately cause faults, in production, while the team is watching. Netflix’s Chaos Monkey became famous for randomly terminating production instances in the middle of the working day, with engineers standing by. The practice is now called chaos engineering: state what “healthy” looks like as a measurable steady state, inject a real-world fault, and check whether the steady state holds, starting with a small blast radius.
Chaos engineering
How much reliability?
Reliability isn’t just for power stations and air-traffic control. Bugs in business software cost people’s time, and worse if they get figures wrong. Outages of a shop cost revenue and reputation. Even “unimportant” apps carry responsibility: imagine a parent who keeps every photo of their kids in your app, and the database gets corrupted.
But reliability has costs, and sometimes you trade it away on purpose: a prototype for an unproven market, or a service with razor-thin margins. That’s a legitimate choice, as long as you know you’re making it.
| Technique | Protects against | What it costs |
|---|---|---|
| Component redundancy (RAID, dual PSUs) | one part dying | extra hardware |
| Multi-machine fault tolerance | a whole machine or VM vanishing | software complexity, more machines |
| Rolling upgrades | planned downtime | requires the row above |
| Timeouts, circuit breakers, load shedding | slow dependencies cascading | tuning; can reject requests that would have succeeded |
| Canary rollouts + fast rollback | a bad change hitting everyone at once | slower deploys, tooling |
| Chaos engineering | recovery paths that silently don’t work | controlled, real risk in production |
| Monitoring and telemetry | not knowing what’s wrong | instrumentation and storage |
Pitfalls
- “We have replicas, so we’re reliable.” Replicas protect against independent faults, like one disk or one machine. They do nothing against a bug or a bad config that hits every replica at once.
- “Slow is better than down.” It’s usually worse: a slow dependency holds callers hostage and cascades. A timeout turns slow into a fast, contained failure.
- “Fault-tolerant means nothing breaks.” It means named faults are contained. Write down which ones.
- Chasing zero faults. You can’t get there. Design for containment and fast recovery instead.
- Trusting the failover you’ve never run. Assume an untested recovery path is broken until a drill proves otherwise.
Recall
Go deeper
- Oppenheimer, Ganapathi & Patterson, “Why Do Internet Services Fail, and What Can Be Done About It?” (USITS, 2003): the outage study behind “operators cause most failures”, with per-service breakdowns.
- Schroeder & Gibson, “Disk Failures in the Real World: What Does an MTTF of 1,000,000 Hours Mean to You?” (FAST, 2007): field replacement rates of 2–4%, against datasheets that imply under 1%.
- Yuan et al., “Simple Testing Can Prevent Most Critical Failures” (OSDI, 2014): why catastrophic failures hide in error-handling code, and how cheap tests find them.
- AWS, “Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region” (2017): one mistyped input, four hours of S3, and the safeguards added afterwards.
- Santosh Janardhan, “More details about the October 4 outage” (Meta, 2021): a capacity check that disconnected every data centre, and an audit tool that failed to stop it.
- Cloudflare, “Understanding how Facebook disappeared from the Internet” (2021): the outside view of the same outage: BGP withdrawals, DNS failing, and the timeline back.
- Izrailevsky & Tseitlin, “The Netflix Simian Army” (2011): the post that introduced Chaos Monkey and its siblings.
- Principles of Chaos Engineering: the five principles, from steady-state hypotheses to minimizing blast radius.
Keep going
The atomic notes for this lesson live in the knowledge graph:
Next on the map: Scalability (what happens as load grows) and Maintainability (keeping the system easy to work on for years).