Notesfoundations🌱 seedling

Service Discovery

Also: Service Registry · Client-Side Discovery · Server-Side Discovery

Service discovery keeps a live registry of which instances of each service exist and where, so balancers and clients always have a current pool. In dynamic environments it becomes the dependency everything depends on.

Why it exists

With autoscaling and short-lived containers, backend addresses change by the minute. Hard-coded IPs go stale, and a stale pool sends traffic to dead or missing instances.

How it works

  • Registration: instances register themselves, or the platform registers them (Kubernetes EndpointSlices). They're removed on drain or missed heartbeats.
  • Server-side discovery: a Load Balancer looks up the pool on the caller's behalf.
  • Client-side discovery: the client fetches the list and balances itself.
  • Mechanisms: DNS (universal, but cached and slow to change), registries built on a Coordination Service (etcd, ZooKeeper, Consul), and push APIs such as Envoy's xDS.
  • Usually combined with Health Checks, so the registry reflects health and not just existence.

Trade-offs

Approach Gains Costs
DNS works with everything TTLs ignored or cached; slow, coarse updates
registry + push near-instant updates, metadata a critical, strongly consistent dependency to operate
client-side no extra hop discovery logic in every client

In practice

  • Roblox, October 2021: a 73-hour outage centred on its Consul cluster, which handled service discovery, health checks, session locking and a key-value store. When Consul degraded, services couldn't find each other, and recovery tooling depended on it too.

Pitfalls

  • Treating discovery as plumbing: it's a Single Point of Failure unless clients can keep working from the last known pool when it's down.
  • Relying on DNS TTLs for fast failover.

Recall

These cards are also in your review deck, scheduled with spaced repetition.