Notesfoundations🌱 seedling
Human Error
Also: Operator Error · Configuration Error
Operators are the leading cause of outages, mostly through configuration mistakes. You can't make humans perfect, but you can make the system forgiving: hard to get wrong, quick to undo, limited in blast radius.
Why it exists
A study of three large internet services found operator error was the leading cause of outages in two of them, mostly configuration mistakes, while hardware caused only 4–25%. Changes to a running system are where people break things.
How it works
- Make the right thing easy: good abstractions, APIs and admin tools. Too restrictive and people work around them.
- Sandboxes: non-production environments with real data, where mistakes are safe.
- Test at every level: unit, integration and manual tests. Automation covers rare corner cases.
- Fast recovery: one-step config rollback, gradual Canary Releases, and tools to recompute bad data.
- Detailed monitoring (telemetry): early warning, and evidence for diagnosis.
- Training and good management.
- Blameless Postmortems: ask how the system allowed it, not who did it.
Trade-offs
| Gains | Costs |
|---|---|
| a mistake hits one canary, not everyone | slower deploys, rollout tooling |
In practice
- Amazon S3, Feb 2017: a mistyped input to a routine command removed far more servers than intended, and
us-east-1was down for hours. - Facebook, Oct 2021: a command meant to assess backbone capacity disconnected every data centre; Facebook was unreachable for nearly six hours.
Pitfalls
- Blaming individuals. That teaches people to hide mistakes instead of fixing the system.
- Pushing changes everywhere at once: a single typo becomes a total outage.
Connections
- Part of
- Reliability
- Learn first
- Fault Tolerance
Recall
These cards are also in your review deck, scheduled with spaced repetition.