Notesfoundations🌱 seedling

Load Shedding

Also: Admission Control · Prioritized Load Shedding · Goodput

Load shedding is a server deliberately rejecting excess work, early and cheaply, so it can spend its capacity on requests it can actually finish. It keeps goodput (useful, timely responses) near capacity instead of letting it collapse.

Why it exists

Past capacity, an unprotected server doesn't level off: queues grow, latency passes clients' timeouts, and it ends up 100% busy producing answers nobody waits for. Once median latency reaches the client timeout, availability is already 50%. When there's more work than capacity something won't get done; shedding decides what, on purpose.

How it works

  • Detect overload from concurrency, queue length or age, or CPU, not raw request rate: requests differ in cost.
  • Reject fast and cheap (HTTP 503, gRPC RESOURCE_EXHAUSTED) before doing real work.
  • Shed by priority: background and prefetch traffic first, user-facing and critical traffic last. Never shed the load balancer's health check.
  • Drop doomed work: a request that already waited past its deadline is wasted; check before starting.
  • Keep queues short, and under overload prefer serving the newest requests (LIFO), since the oldest are most likely abandoned.
  • Clients can shed too: Google's adaptive throttling rejects locally with probability max(0, (requests − K·accepts)/(requests + 1)), K = 2.

Trade-offs

Gains Costs
goodput stays near capacity some requests fail outright
server stays alive, no crash cascade needs priorities and good overload signals
fast errors let clients react misleadingly good latency metrics while shedding

In practice

  • Netflix (2024): playback requests a user waits on get guaranteed capacity; prefetch gets the rest. During a 12× prefetch spike, user-initiated availability stayed above 99.4%.
  • Google: every request carries a criticality (CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE).
  • Uber Cinnamon: a PID controller sets the rejection rate automatically, across 768 priority levels.

Pitfalls

  • Shedding health checks, so the load balancer marks the server dead and moves its load onto others.
  • Never load-testing past capacity, so nobody knows whether goodput plateaus or collapses.

Recall

These cards are also in your review deck, scheduled with spaced repetition.