Notesdistributed🌱 seedling
Process Pause
Also: GC Pause · Stop-the-World Pause · Stop-the-World
A running thread can be stopped at any instruction for seconds or minutes without knowing it, while the rest of the system moves on. Any "check, then act" based on time is unsafe across such a pause.
Why it exists
Runtimes, operating systems and hypervisors all preempt code: a node can't assume its own execution is continuous.
How it works
- Causes: stop-the-world garbage collection (JVM pauses have lasted minutes); VM suspension and live migration; hypervisor steal time; swapping to disk; synchronous I/O on a slow disk; SIGSTOP; a laptop lid.
- The bug pattern: a node checks its Lease ("10 s left"), pauses for 15 s, wakes, and acts as if it still holds the lease, while another node has taken over.
- Why checking harder doesn't help: the pause can fall right after the check, and the paused code sees no gap.
- Defences: don't rely on the actor's sense of time; make the resource reject stale actors with a Fencing Token; treat a paused node like a briefly dead one.
Trade-offs
| Mitigation | Gains | Costs |
|---|---|---|
| low-pause GC (e.g. concurrent collectors), no swap | shorter pauses | more CPU or memory; never zero pauses |
| planned GC (drain traffic before collecting) | pauses hidden from clients | runtime and load-balancer cooperation |
| fencing at the resource | correct regardless of pause length | every resource must check tokens |
In practice
- HBase used to have this bug: a region server could pause past its ZooKeeper session expiry and keep writing after its regions were reassigned.
- Lease-based leaders (in Leader-Based Replication and Consensus systems) step down only when they notice; a paused leader notices late.
Pitfalls
- "Our GC pauses are short." The tail matters: one long pause in a million is enough to corrupt data.
- Real-time guarantees exist (RTOS, avionics) but at great cost; typical servers have none.
Connections
- Part of
- Distributed Systems
- Builds on this
- Fencing Token
- Mentioned
- Leader-Based ReplicationConsensus
Recall
These cards are also in your review deck, scheduled with spaced repetition.