Notesfoundations🌱 seedling
Rate Limiting
Also: Rate Limiter · Throttling · Token Bucket · Quota
Rate limiting caps how much any one caller (user, API key, tenant) may send per unit of time, rejecting the excess with 429 Too Many Requests. It protects shared capacity from one client, where load shedding protects a server from total load.
Why it exists
Without per-caller limits, one buggy script, retry loop or abusive client can consume capacity meant for everyone else. Limits also make capacity planning possible: you know the worst a caller can do.
How it works
- Token bucket (most common): tokens refill at the allowed rate up to a burst size; each request spends one. Allows short bursts, enforces the average.
- Sliding-window counters: count requests in a rolling window; tighter control over bursts.
- Concurrency limits: cap requests in flight per caller, which also catches expensive slow requests that a rate limit misses.
- Return
429with aRetry-Afterheader so well-behaved clients back off (Retry with Backoff). - Distributed limits need shared state (often Redis) or approximate per-node limits.
Trade-offs
| Gains | Costs |
|---|---|
| fairness between callers | legitimate bursts get rejected |
| bounded worst case per client | shared limiter state is itself a dependency |
| abuse and runaway loops contained | limits need real usage data to set |
In practice
- Stripe layers four limiters: per-user request rate (token buckets in Redis), per-user concurrent requests (e.g. 20 in flight), a fleet-wide shedder that reserves 20% of capacity for critical requests, and a last-resort shedder that drops traffic by category.
- Their launch advice: dark-launch each limiter, fail open if the limiter's store is down, and keep a kill switch.
Pitfalls
- A limiter that fails closed when its Redis is down takes the whole API down with it.
- Limits set by guesswork, never observed in shadow mode first.
Connections
- Part of
- Foundations
- In real systems
- Redis
Recall
These cards are also in your review deck, scheduled with spaced repetition.