Notesfoundations🌱 seedling

Retry with Backoff

Also: Exponential Backoff · Jitter · Full Jitter · Retry Budget · Retry Storm

Retrying hides transient faults, but every retry is extra load that arrives exactly when a dependency is struggling. Safe retries wait exponentially longer with random jitter, happen at one layer, and stay within a budget.

Why it exists

Most distributed failures are transient: a dropped packet, a restarting server, a brief throttle. Retrying turns them into a little extra latency instead of an error. Done carelessly, it multiplies load and sustains outages (Metastable Failure).

How it works

  • Amplification: retries multiply across layers. Five layers that each try 3 times put up to 3⁵ = 243 attempts on the bottom layer for one request.
  • Retry at one layer, usually the one nearest the failure; others fail fast. An overloaded backend can return an explicit "don't retry" error.
  • Budget: cap attempts per request (gRPC caps maxAttempts at 5) and cap retries as a share of traffic. A per-request limit of 3 still allows nearly 3× load; a 10% retry budget caps it at 1.1×.
  • Exponential backoff with full jitter: sleep random(0, min(cap, base × 2ⁿ)). Without jitter, clients that failed together retry together, in synchronised waves.
  • Token bucket: a retry spends tokens, a success earns some back; an empty bucket means no retries in the AWS SDKs; gRPC's retry throttling pauses retries once the bucket falls below half.
  • Only retry operations that are safe to repeat: reads, or writes made idempotent with an idempotency key.

Trade-offs

Gains Costs
masks transient faults extra load during outages
fewer user-visible errors added latency; needs idempotent APIs
jitter smooths load slightly longer worst-case completion

In practice

  • AWS SDKs: standard mode defaults to 3 attempts, exponential backoff with full jitter capped at 20 s, and a 500-token retry quota.
  • Envoy: a retry budget defaulting to 20% of active requests overrides the fixed retry limit.

Pitfalls

  • Exponential backoff without jitter: a series of synchronised spikes.
  • Every layer retrying "just 3 times".
  • Retrying non-idempotent writes: double charges, duplicate orders.

Recall

These cards are also in your review deck, scheduled with spaced repetition.