Notesdistributed🌱 seedling
Unreliable Network
Also: Asynchronous Network · Network Faults · Fallacies of Distributed Computing · Unbounded Delay
Data-centre and internet networks are asynchronous packet networks: a message may be lost, delayed without bound, duplicated or reordered, and the sender can't tell a lost request from a dead or slow receiver or a lost reply.
Why it exists
Shared networks give high utilization by queueing packets instead of reserving capacity per connection (like a phone circuit would). Queues mean variable delay; full queues mean drops. That bargain buys cheap bandwidth at the price of no timing guarantees.
How it works
- Where delay comes from: a switch port queue when many senders target one receiver; the receiving OS when every core is busy; a hypervisor running another VM; TCP retransmitting a lost packet after its timeout (at least 200 ms on Linux), which turns loss into latency.
- What the sender sees: silence, whatever went wrong. The only tool is a Timeout, which is a guess about a delay distribution.
- What else can happen: duplicates (a retry after a lost reply), reordering (a retransmitted packet arriving after a later one), and Network Partitions.
- TCP gives ordering and retransmission within one connection, not delivery guarantees across reconnects or processes.
Trade-offs
| Network design | Gains | Costs |
|---|---|---|
| packet switching with queues (Ethernet, IP) | high utilization, cheap bursts | unbounded delay and loss |
| circuit-like reserved capacity | bounded latency | idle capacity is wasted |
In practice
- Microsoft data centres (2011 study): on average 5.2 device failures and 40.8 link failures per day; redundant paths cut the median impact of a failure by only 40%.
- GitHub, December 2012: a switch fault blocked traffic for 90 seconds; delayed fencing messages left some file-server pairs both active, and on recovery each shot the other.
Pitfalls
- Assuming "the network is reliable" and "latency is zero" (the first two Fallacies of Distributed Computing).
- Redundant switches that share software and configuration fail together.
Connections
- Part of
- Distributed Systems
- Contains
- Network Partition
- Builds on this
- Failure Detection
Recall
These cards are also in your review deck, scheduled with spaced repetition.