Notesdistributed🌱 seedling

Network Partition

Also: Netsplit · Partial Partition · Partial Network Partition · Simplex Partition

A network fault that cuts some nodes off from others while all of them keep running. To each side, the other side looks crashed, so a system must ensure at most one side keeps making decisions.

Why it exists

Switches, links, NICs and routing configs fail; one top-of-rack switch isolates a whole rack. The nodes are healthy, so they keep acting on whatever they can see.

How it works

  • Complete partition: two groups, no traffic between them.
  • Partial partition: A and B can't talk, but C talks to both, so nodes disagree about who is up.
  • Simplex partition: traffic flows one way only (a node's heartbeats get out, requests don't get in).
  • Safe handling: only a side holding a majority (Quorum) may elect a leader or commit writes; the minority side stops writing or serves stale reads explicitly. Without this you get Split Brain.
  • After healing, nodes must reconcile: step down from stale leadership, replay or discard divergent writes.

Trade-offs

During a partition Gains Costs
minority refuses writes (CP) no divergent histories unavailable on the minority side
both sides accept writes (AP) available everywhere conflicts to resolve later (Write Conflict)

In practice

  • Partition-failure study (OSDI 2018): 136 failures in 25 systems; 80% catastrophic (data loss most common), 88% triggerable by isolating one node, 29% caused by partial partitions, leader election the most affected mechanism.
  • Cloudflare, November 2020: a half-failed switch split one etcd node from the leader while a third node reached both; repeated elections left etcd read-only for six minutes and the API impaired for 6 h 33 min.

Pitfalls

  • Testing only clean, complete partitions. Partial and one-way partitions find the bugs.
  • Two-node clusters: neither side can form a majority.

Recall

These cards are also in your review deck, scheduled with spaced repetition.