Notesfoundations🌱 seedling

Health Check

Also: Health Checking · Outlier Detection · Passive Health Check · Fail Open · Panic Threshold

A health check decides whether a backend should get traffic right now. Active checks probe an endpoint on a timer. Passive checks (outlier detection) watch real responses. Both need guardrails so a correlated failure doesn't empty the pool.

Why it exists

Backends die, hang or break in ways the process itself can't see. Without a health signal the balancer keeps sending traffic into the hole.

How it works

  • Active: probe /health every interval. Remove after N consecutive failures; re-admit after M passes (hysteresis). Detection lag is about interval × N.
  • Shallow vs deep: a shallow check asks "can this server serve?" (process up, disk writable). A deep check also tests dependencies, so one dependency blip fails every server at once.
  • Passive / outlier detection: count real errors and eject a backend for a growing period after, say, 5 consecutive 5xx. Cap ejections at a small share of the pool.
  • Fail open: if too much of the pool looks unhealthy, assume the signal is wrong and send traffic to everyone. A total outage is worse than a partial one.

Trade-offs

Choice Gains Costs
faster probes, lower thresholds quicker removal probe load, flapping
deep checks catch broken credentials, config correlated mass ejection
passive checks test the real work need traffic; can eject for a bad client

In practice

  • AWS ALB: defaults to a probe every 30 s, unhealthy after 2 failures, healthy after 5 passes. It fails open, routing to all targets if all are unhealthy.
  • Envoy: outlier detection ejects after 5 consecutive 5xx for 30 s (longer on repeats), at most 10% of hosts. Below 50% healthy hosts (the panic threshold) it ignores health entirely.

Pitfalls

  • The black hole: a server that fails fast still passes /health, and a load-aware algorithm sends it the most traffic. Passive checks catch it.
  • Deep checks on the balancer turn a dependency hiccup into a full outage. Send dependency checks to monitoring instead.

Recall

These cards are also in your review deck, scheduled with spaced repetition.