Notesfoundations🌱 seedling
Service Discovery
Also: Service Registry · Client-Side Discovery · Server-Side Discovery
Service discovery keeps a live registry of which instances of each service exist and where, so balancers and clients always have a current pool. In dynamic environments it becomes the dependency everything depends on.
Why it exists
With autoscaling and short-lived containers, backend addresses change by the minute. Hard-coded IPs go stale, and a stale pool sends traffic to dead or missing instances.
How it works
- Registration: instances register themselves, or the platform registers them (Kubernetes EndpointSlices). They're removed on drain or missed heartbeats.
- Server-side discovery: a Load Balancer looks up the pool on the caller's behalf.
- Client-side discovery: the client fetches the list and balances itself.
- Mechanisms: DNS (universal, but cached and slow to change), registries built on a Coordination Service (etcd, ZooKeeper, Consul), and push APIs such as Envoy's xDS.
- Usually combined with Health Checks, so the registry reflects health and not just existence.
Trade-offs
| Approach | Gains | Costs |
|---|---|---|
| DNS | works with everything | TTLs ignored or cached; slow, coarse updates |
| registry + push | near-instant updates, metadata | a critical, strongly consistent dependency to operate |
| client-side | no extra hop | discovery logic in every client |
In practice
- Roblox, October 2021: a 73-hour outage centred on its Consul cluster, which handled service discovery, health checks, session locking and a key-value store. When Consul degraded, services couldn't find each other, and recovery tooling depended on it too.
Pitfalls
- Treating discovery as plumbing: it's a Single Point of Failure unless clients can keep working from the last known pool when it's down.
- Relying on DNS TTLs for fast failover.
Connections
- Part of
- Load Balancer
- Mentioned
- Single Point of Failure
Recall
These cards are also in your review deck, scheduled with spaced repetition.