Notesfoundations🌱 seedling
Chaos Engineering
Also: Fault Injection
Deliberately inject faults into production (kill instances, cut links) while the team is ready, so recovery paths are exercised continuously instead of failing for the first time in a real outage.
Why it exists
Many critical bugs live in error-handling code (failover, retries, recovery) that rarely runs and quietly rots.
How it works
- Define steady state as a measurable output users care about (e.g. successful plays per second), not internal metrics.
- Hypothesize it holds under a fault you claim to tolerate (e.g. "any instance can die").
- Inject a real-world fault (kill an instance, add latency, cut a zone) in production during working hours, with monitoring on.
- If steady state breaks, you've found a gap in Fault Tolerance. Fix it before a real outage finds it at 3 a.m.
- Automate the experiments so they keep running as the system changes.
Trade-offs
| Gains | Costs |
|---|---|
| recovery paths stay proven | controlled but real risk in production |
In practice
- Chaos Monkey (Netflix, 2011) randomly terminates production instances; its siblings in the Simian Army took out whole availability zones and regions.
- Fault-injection services from cloud providers now run the same experiments as managed tooling.
Pitfalls
- Starting with large blast radii before small experiments pass.
- Running it with no monitoring or rollback plan.
Connections
- Part of
- Fault Tolerance
- In real systems
- Chaos MonkeyJepsen
Recall
These cards are also in your review deck, scheduled with spaced repetition.