Notesfoundations🌱 seedling

Rate Limiting

Also: Rate Limiter · Throttling · Token Bucket · Quota

Rate limiting caps how much any one caller (user, API key, tenant) may send per unit of time, rejecting the excess with 429 Too Many Requests. It protects shared capacity from one client, where load shedding protects a server from total load.

Why it exists

Without per-caller limits, one buggy script, retry loop or abusive client can consume capacity meant for everyone else. Limits also make capacity planning possible: you know the worst a caller can do.

How it works

  • Token bucket (most common): tokens refill at the allowed rate up to a burst size; each request spends one. Allows short bursts, enforces the average.
  • Sliding-window counters: count requests in a rolling window; tighter control over bursts.
  • Concurrency limits: cap requests in flight per caller, which also catches expensive slow requests that a rate limit misses.
  • Return 429 with a Retry-After header so well-behaved clients back off (Retry with Backoff).
  • Distributed limits need shared state (often Redis) or approximate per-node limits.

Trade-offs

Gains Costs
fairness between callers legitimate bursts get rejected
bounded worst case per client shared limiter state is itself a dependency
abuse and runaway loops contained limits need real usage data to set

In practice

  • Stripe layers four limiters: per-user request rate (token buckets in Redis), per-user concurrent requests (e.g. 20 in flight), a fleet-wide shedder that reserves 20% of capacity for critical requests, and a last-resort shedder that drops traffic by category.
  • Their launch advice: dark-launch each limiter, fail open if the limiter's store is down, and keep a kill switch.

Pitfalls

  • A limiter that fails closed when its Redis is down takes the whole API down with it.
  • Limits set by guesswork, never observed in shadow mode first.

Recall

These cards are also in your review deck, scheduled with spaced repetition.