NotesSystem
Jepsen
Also: Knossos · Elle
Kyle Kingsbury's framework and series of analyses (since 2013) that test distributed databases under injected faults and check the recorded histories against the consistency models they claim.
Also in: Serializability › In the real world · Consistency Models › In the real world
Design choices
- Black-box histories: clients run operations against a real cluster and record every invocation and result, including timeouts as "unknown".
- Fault injection: network partitions (including partial ones), process pauses and crashes, clock skew, membership changes. See Network Partition and Clock Skew.
- Checkers: Knossos searches for a valid linearization; Elle (2020) infers transaction dependency graphs to find isolation anomalies efficiently.
- The consistency map: jepsen.io/consistency relates the models and marks each as totally available, sticky available, or unavailable under partition.
Notable findings
- etcd 3.4.3 (2020): key-value operations strict serializable; locks unsafe even in healthy clusters, with about 18% of acknowledged updates lost in one lock-protected test. Fix: fence with the lock key's revision.
- jetcd 0.8.2 (2024): the Java client retried non-idempotent requests that may have succeeded, causing lost updates; etcd itself wasn't at fault.
- NATS 2.12.1 (2025): flushed to disk every two minutes by default while acknowledging immediately; docs claimed linearizable and always available.
Notable trade-offs
- Testing finds bugs, it can't prove their absence: a passing run means no violation was observed under those faults.
- Many analyses are vendor-funded, published under a stated ethics policy.
Connections
- Related
- Consistency Model
- Mentioned
- Clock Skewetcd