Notesfoundations🌱 seedling

Chaos Engineering

Also: Fault Injection

Deliberately inject faults into production (kill instances, cut links) while the team is ready, so recovery paths are exercised continuously instead of failing for the first time in a real outage.

Why it exists

Many critical bugs live in error-handling code (failover, retries, recovery) that rarely runs and quietly rots.

How it works

  • Define steady state as a measurable output users care about (e.g. successful plays per second), not internal metrics.
  • Hypothesize it holds under a fault you claim to tolerate (e.g. "any instance can die").
  • Inject a real-world fault (kill an instance, add latency, cut a zone) in production during working hours, with monitoring on.
  • If steady state breaks, you've found a gap in Fault Tolerance. Fix it before a real outage finds it at 3 a.m.
  • Automate the experiments so they keep running as the system changes.

Trade-offs

Gains Costs
recovery paths stay proven controlled but real risk in production

In practice

  • Chaos Monkey (Netflix, 2011) randomly terminates production instances; its siblings in the Simian Army took out whole availability zones and regions.
  • Fault-injection services from cloud providers now run the same experiments as managed tooling.

Pitfalls

  • Starting with large blast radii before small experiments pass.
  • Running it with no monitoring or rollback plan.

Recall

These cards are also in your review deck, scheduled with spaced repetition.