Notesfoundations🌱 seedling
Retry with Backoff
Also: Exponential Backoff · Jitter · Full Jitter · Retry Budget · Retry Storm
Retrying hides transient faults, but every retry is extra load that arrives exactly when a dependency is struggling. Safe retries wait exponentially longer with random jitter, happen at one layer, and stay within a budget.
Why it exists
Most distributed failures are transient: a dropped packet, a restarting server, a brief throttle. Retrying turns them into a little extra latency instead of an error. Done carelessly, it multiplies load and sustains outages (Metastable Failure).
How it works
- Amplification: retries multiply across layers. Five layers that each try 3 times put up to 3⁵ = 243 attempts on the bottom layer for one request.
- Retry at one layer, usually the one nearest the failure; others fail fast. An overloaded backend can return an explicit "don't retry" error.
- Budget: cap attempts per request (gRPC caps
maxAttemptsat 5) and cap retries as a share of traffic. A per-request limit of 3 still allows nearly 3× load; a 10% retry budget caps it at 1.1×. - Exponential backoff with full jitter: sleep
random(0, min(cap, base × 2ⁿ)). Without jitter, clients that failed together retry together, in synchronised waves. - Token bucket: a retry spends tokens, a success earns some back; an empty bucket means no retries in the AWS SDKs; gRPC's retry throttling pauses retries once the bucket falls below half.
- Only retry operations that are safe to repeat: reads, or writes made idempotent with an idempotency key.
Trade-offs
| Gains | Costs |
|---|---|
| masks transient faults | extra load during outages |
| fewer user-visible errors | added latency; needs idempotent APIs |
| jitter smooths load | slightly longer worst-case completion |
In practice
- AWS SDKs: standard mode defaults to 3 attempts, exponential backoff with full jitter capped at 20 s, and a 500-token retry quota.
- Envoy: a retry budget defaulting to 20% of active requests overrides the fixed retry limit.
Pitfalls
- Exponential backoff without jitter: a series of synchronised spikes.
- Every layer retrying "just 3 times".
- Retrying non-idempotent writes: double charges, duplicate orders.
Connections
- Part of
- Foundations
- Learn first
- Timeout
- Builds on this
- Metastable Failure
- Related
- Circuit BreakerDead Letter QueueDeadlockDelivery SemanticsDurable ExecutionEnd-to-End ArgumentIdempotencyRate LimitingTail Latency AmplificationUnreliable Network
- In real systems
- Temporal
Recall
These cards are also in your review deck, scheduled with spaced repetition.