Timeouts, Retries & Overload
Most big outages aren't caused by something breaking. They're caused by everything else reacting to it. How to give up gracefully, retry without stampeding, and stay useful when there's more work than capacity.
The whole idea
Why overload is different
Most faults are local: a disk dies, a node gets routed around. Overload is different, because the system’s own reactions make it worse. A server that’s a little too busy gets slower. Slower answers make clients time out. Clients that time out send the request again. Now the server is much too busy, and it’s spending its effort on answers nobody is waiting for any more.
Two numbers make this precise. Throughput is every request the server handles per second. Goodput is the subset that succeeds, quickly enough to be useful. Below capacity they’re the same. Above it, they part ways, and how far they part depends entirely on how the server behaves when it can’t keep up.
What happens past capacity
The red curve isn’t a strawman. Once a server’s median latency reaches its clients’ timeout, half its responses arrive too late, so its availability is 50% even though it’s working flat out. A little more load and almost nothing it produces is useful. That’s congestive collapse: a busy server doing no useful work.
Timeouts
Every call across a network can hang: the remote process is paused, a packet vanished, a lock is held. Without a timeout, the caller waits forever, holding a thread, a connection and some memory. Enough of those and the caller is out of resources too. A slow dependency is usually more dangerous than a dead one, because dead fails fast and slow holds everyone hostage.
Choosing the number is a trade-off between two errors. Too long, and stuck calls tie up resources. Too short, and you abandon calls that would have succeeded, and probably retry them, adding load. A good recipe from AWS: decide what rate of false timeouts you’ll accept, say 0.1%, and set the timeout at the matching latency percentile of the dependency, here its p99.9. Then pad it for network variance.
A single timeout per call isn’t enough in a chain of services. If the client gives up after 1 second but each hop has its own 30-second default, the servers keep working on a request whose caller left long ago. The fix is a deadline: an absolute budget set once, at the edge, and passed along with the request. Each hop spends some of it and forwards only what’s left.
Deadlines travel with the request
gRPC builds this in: a client sets a deadline, the remaining time is sent to the server as a timeout (which avoids
clock skew), and when it expires the client gets DEADLINE_EXCEEDED while the server sees the call cancelled.
Java and Go propagate deadlines to downstream calls automatically. But by default gRPC sets no deadline at
all, and the same is true of many HTTP clients. A missing timeout is the most common resilience bug there is,
because it’s invisible until the day it matters.
Retries and backoff
Most failures in a distributed system are transient: a dropped packet, a server mid-restart, a brief throttle. Retrying hides them, which is why every client library does it. But retries are also load, and they arrive at the worst possible moment: exactly when the thing being retried is struggling.
The danger multiplies with depth. If every layer of a stack retries independently, the attempts multiply at each level.
Retry amplification
The fixes, in order of importance:
- Retry at one layer. Usually the one closest to the failure, or the edge. Every other layer fails fast and passes the error up. Google’s guidance is to retry only at the layer directly above the one rejecting, and for an overloaded backend to return an explicit “overloaded, don’t retry” error that stops retries further up.
- Budget retries. Cap attempts per request (gRPC caps
maxAttemptsat 5), and cap retries as a fraction of traffic. A per-request limit of 3 still allows almost 3× load during an outage. A budget allowing retries only while they stay under 10% of requests caps it at 1.1×. Envoy’s retry budget defaults to 20% of active requests. The AWS SDKs implement the same idea as a token bucket: a retry spends tokens, a success earns some back, and when the bucket is empty, the SDK stops retrying.The AWS SDKs updated their standard retry mode in 2026: a transient-error retry now costs 14 of the bucket’s 500 tokens, a throttling retry 5. The change became the default on 10 November 2026 in every SDK except Kotlin. - Back off, with jitter. Wait longer after each failure, exponentially, up to a cap. And add randomness, or every client that failed together will retry together.
- Only retry what’s safe. A retried payment is a double payment unless the API is idempotent. A request carrying a unique idempotency key can be applied at most once however many times it’s sent.
Backoff without jitter is a trap. The waits are deterministic, so clients that failed at the same instant (which is what an outage does) compute identical schedules and come back as a synchronised wave.
Why backoff needs jitter
AWS’s comparison of the variants found plain exponential backoff “the clear loser”. Full jitter, sleeping a
random time between zero and min(cap, base × 2ⁿ), did the least total work while finishing almost as fast as
anything else. The same logic applies beyond retries: any periodic job or timer started on many machines at once
deserves jitter.
Cascading failures
A cascading failure is one where a failure in one part raises the load on, or starves, the parts around it, so they fail too. The classic shape is load redistribution: when one server in a pool dies, its share moves to the survivors, which pushes them closer to their own limit.
How a pool falls over one server at a time
What actually kills an overloaded server is usually resource exhaustion. When the CPU runs out, every request gets slower, so more are in flight at once, which uses more memory. When memory runs out, tasks die or caches shrink, so hit rates fall and backends get hammered. Garbage-collected runtimes have a special version: less free memory means more GC, which means less CPU, which means slower requests and yet more memory in use. Running out of threads or file descriptors stops a server from accepting connections, including health checks, so the load balancer declares it dead and moves its load elsewhere.
Servers with cold caches or slow startup can’t take full traffic the moment they’re back, so restarts need traffic ramped up gradually. And capacity can fail in surprising ways: in November 2020, adding servers to Amazon Kinesis’s front-end fleet pushed every server past an operating system limit on threads, because each server kept a thread for every other server in the fleet. Adding capacity triggered the outage; the thread limit was the root cause. It cascaded to Cognito, CloudWatch, Lambda and EventBridge, and full recovery took until late that evening, partly because servers could only be safely brought back a few hundred per hour.
Metastable failures
Some outages outlive their cause. A metastable failure is one where a temporary trigger pushes a system into a bad state that persists after the trigger is gone, held there by a feedback loop in the system itself. The term comes from a 2021 HotOS paper by Bronson, Aghayev, Charapko and Zhu, which describes three states:
- Stable: load is low enough that the system recovers from a hiccup on its own.
- Vulnerable: healthy and efficient, but above an invisible threshold. Many systems run here deliberately, because it’s cheaper. It isn’t overload; it just isn’t safe against a trigger.
- Metastable: goodput is near zero, and a sustaining effect (usually work amplification, such as retries) keeps it there.
An outage with no cause
The paper’s own example: a database answers in under 100 ms up to 300 queries a second, and an application retries once after a 1-second timeout. Running at 280 queries a second, a 10-second network blip makes requests time out. Their retries double the load to 560 a second, which keeps requests timing out, which keeps retries coming. The system only recovers if load drops below 150 a second, or retries are throttled hard. A system running happily at 280 can’t get back from the blip on its own.
The 2022 follow-up study, Metastable Failures in the Wild, found that load shedding was the most common mitigation, used in over half of the incidents, and that outages are typically blamed on the trigger when the true root cause is the sustaining effect. That matters for postmortems: fixing the trigger (the bad deploy, the network blip) leaves the loop in place for the next trigger.
A recent, vivid example: on 20 October 2025, a race condition in DynamoDB’s DNS automation left its us-east-1
endpoint with an empty DNS record. DNS was restored by 2:25 a.m. Pacific. But the EC2 subsystem that manages leases
on the physical hosts behind instances had lost those leases meanwhile, and when it tried to re-establish them all,
the attempts took so long they timed out and were queued again. AWS’s write-up says it “had entered a state of
congestive collapse”. It took engineers throttling incoming work and selectively restarting the lease-manager hosts,
from 4:14 a.m., to break the loop. The trigger was gone for nearly two hours before recovery even began.
Circuit breakers and bulkheads
Retries assume failures are brief. When a dependency is properly down, retrying it just adds load and wastes the caller’s time. A circuit breaker wraps calls to a dependency and watches their failure rate. When failures cross a threshold, it trips, and later calls fail immediately without being sent.
Circuit breaker states
Breakers protect both sides: the caller keeps its threads and can serve a fallback (cached results, a degraded page), and the dependency gets room to recover. The cost is tuning: a breaker that trips too eagerly turns a blip into a minutes-long outage of that feature, and one shared breaker in front of a sharded backend can cut off healthy shards because one is sick.
Netflix’s Hystrix popularised breakers in Java and is now in maintenance mode; its README points new projects to resilience4j and to adaptive approaches that react to measured latency rather than fixed thresholds. Service meshes do the job at the network layer. Envoy’s “circuit breakers” are really concurrency caps per upstream cluster (1,024 connections, pending requests and active requests by default, 3 concurrent retries), and its outlier detection ejects an individual host after 5 consecutive 5xx responses, for 30 seconds at first.
A bulkhead is the compartment idea from ship hulls: partition resources so that one flooding compartment doesn’t sink the ship. Give each dependency its own thread or connection pool, and a slow recommendations service can exhaust only the recommendations pool, while checkout keeps its own. At fleet scale, the same idea becomes cells and shuffle sharding. Route 53 spreads customers across 2,048 virtual name servers, giving each domain a “shuffle shard” of four. There are about 730 billion possible shards, and no two customer domains share more than two name servers, so a flood of traffic aimed at one customer can’t take out another.
Load shedding and backpressure
When there’s more work than capacity, something won’t get done. The only choice is whether that’s decided on purpose, cheaply and early, or by accident, after the work has been half-done and the client has left.
Load shedding is deciding on purpose. A server that knows it’s at its limit rejects excess requests immediately, with a cheap error, and spends its capacity on the requests it can actually complete. Done well, it is what turns the red curve into the green one. The details that matter:
- Shed by priority. Not all traffic is equal. At Google every request carries a criticality (
CRITICAL_PLUS,CRITICAL,SHEDDABLE_PLUS,SHEDDABLE) and sheds from the bottom. The load balancer’s health check is the most important request a server gets: shed it and the server is declared dead, which moves its load onto everyone else. - Drop doomed work. If a request has already waited in a queue past its client’s deadline, any work on it is wasted. Check the deadline before starting.
- Keep queues short. A queue exists to absorb bursts, not overload. Google suggests queues no longer than about half the thread pool. Under overload, serving the newest request first (LIFO) beats FIFO, because the oldest requests are the ones most likely to have been abandoned.
Shedding by priority
Clients can shed too. In Google’s adaptive throttling, each client tracks its recent requests and how many the
backend accepted, and starts rejecting locally with probability max(0, (requests − K × accepts) / (requests + 1)).
With K = 2, a client still sends up to about twice what the backend has been accepting, so the backend keeps
seeing (and cheaply rejecting) some excess. Google chooses that deliberately: wasting a little backend effort
lets the client notice quickly when capacity comes back.
Backpressure is the other answer to the same problem: instead of rejecting work, the consumer tells the
producer to slow down. TCP flow control does it with its receive window. Reactive Streams does it with
request(n). A pull-based consumer, like a Kafka consumer reading at its own pace, gets it for free. Backpressure
loses nothing, which is why it suits pipelines and batch work. But it only works if the producer can afford to
wait. A web user can’t be told to slow down; at the edge, the pressure has to turn into a rejection somewhere.
Rate limiting
Load shedding protects a server from total load. Rate limiting protects it, and its other customers, from
any one caller: each client, API key or tenant gets a quota, and requests beyond it are rejected with a 429 Too Many Requests, ideally with a Retry-After header. The usual algorithm is a token bucket: tokens drip in at
the allowed rate up to a burst size, and each request spends one. It allows short bursts while enforcing an
average rate. Sliding-window counters are the common alternative when bursts must be bounded tightly.
Stripe’s 2017 write-up describes a layered setup that’s still a good template: per-user token buckets in Redis, a per-user cap on concurrent requests, a fleet-wide shedder that reserves 20% of capacity for critical requests, and a last-resort shedder that drops traffic by category. Its operational advice matters as much: dark-launch each limiter, make it fail open if its own store is down, and keep a kill switch.
Trade-offs
| Defence | Protects against | Costs | Gets it wrong by |
|---|---|---|---|
| Timeouts and deadlines | slow dependencies holding resources | tuning per call | too short: false failures and extra retries |
| Retries with backoff and jitter | transient faults | added load, latency, idempotency work | retrying at every layer, or without a budget |
| Circuit breaker | hammering a dependency that’s down | fallback logic, tuning | tripping on blips; one breaker for many shards |
| Bulkhead / cells | one dependency or tenant exhausting everything | idle reserved capacity, more partitions | pools too small to absorb normal bursts |
| Load shedding | collapse under excess load | some requests fail | shedding health checks or critical traffic |
| Backpressure | fast producers swamping consumers | producers slow down | applied where callers can’t wait |
| Rate limiting | one caller hogging shared capacity | legitimate bursts get 429s | limits set without measuring real usage |
In the real world
- Google Cloud, June 12, 2025. A policy change with blank fields hit a new code path in Service Control that
had no error handling and no feature flag, and the binary crash-looped worldwide. The fix was out within about
40 minutes, but in
us-central1the restarting tasks “created a herd effect” on the Spanner table they depend on, because Service Control “did not have the appropriate randomized exponential backoff implemented”. That region took up to about 2 hours 40 minutes to recover. - AWS, December 7, 2021. An automated capacity-scaling activity triggered a surge of connection attempts from clients inside AWS’s internal network, overwhelming the devices linking it to the main network. Errors caused “even more connection attempts and retries”. AWS’s summary notes that “a latent issue prevented these clients from adequately backing off”.
- Slack, February 22, 2022. A rolling Consul upgrade emptied part of the memcached fleet. Cache misses for one query became scatter queries across every database shard, which timed out, which meant the cache could never be refilled. Slack’s write-up explicitly frames it as a metastable failure. The way out was throttling client boots and then raising the limit gradually; their first attempt raised it too fast.
- Netflix, 2024. Netflix’s playback API sheds by priority inside the service: requests a user is waiting on are guaranteed capacity, and pre-fetch requests only get what’s left. During one infrastructure outage, Android pre-fetch traffic spiked 12×. Pre-fetch availability fell as low as 20%, while user-initiated requests stayed above 99.4%.
- Uber, 2023. Uber’s Cinnamon load shedder needs no per-service configuration: a PID controller sets the rejection rate to keep queues short, a TCP-Vegas-style tuner sets the concurrency limit, and requests are ranked into 768 priority levels, with users grouped into cohorts so the same users get shed consistently across services.
Pitfalls
- No timeout at all. The default in many clients. Grep for it; it’s the most common resilience bug in production code.
- Retries at every layer. Each one looks harmless. Together they multiply. Pick one layer and give it a budget.
- Exponential backoff without jitter. Synchronised clients turn one outage into a series of them.
- Blaming the trigger. If an outage outlasted its cause, the root cause is the loop that sustained it. Find it.
- Shedding the health check. Reject the work, never the ping, or the load balancer moves your load onto everyone else.
- Unbounded queues. They don’t absorb overload; they postpone it, and fill with work nobody wants.
- Untested limits. A limiter or breaker that’s never fired in production is a guess. Load-test past capacity and watch what goodput does.
Recall
Go deeper
- Timeouts, retries, and backoff with jitter (Marc Brooker, AWS Builders’ Library, 2019). The practical rules: picking timeouts from percentiles, retrying at one layer, token-bucket retries.
- Exponential Backoff And Jitter (Marc Brooker, AWS Architecture Blog, 2015). The simulation behind full jitter, now the AWS SDKs’ default.
- Using load shedding to avoid overload (David Yanacek, AWS Builders’ Library, 2019). Goodput, prioritisation, and dropping doomed work.
- Handling Overload and Addressing Cascading Failures (Google SRE). Criticality, adaptive throttling, retry budgets, and the anatomy of a cascade.
- Metastable Failures in Distributed Systems (Bronson et al., HotOS 2021). Short, and it names the thing you’ve probably already lived through.
- Metastable Failures in the Wild (Huang et al., OSDI 2022). 22 real incidents dissected: triggers, sustaining effects, what worked.
- Enhancing Netflix Reliability with Service-Level Prioritized Load Shedding (Netflix, 2024). Priority shedding inside a service, with production numbers.
- AWS us-east-1 post-event summary, October 2025. A modern metastable failure, told in timestamps.
Keep going
The atomic notes for this lesson live in the knowledge graph:
Next on the map: Load Balancing (spreading the load these defences protect) and Caching (where cache stampedes are overload in disguise).