Notesfoundations🌱 seedling

Observability

Also: O11y · Distributed Tracing · Telemetry · Wide Events

Observability is the ability to understand what a system is doing, and answer questions you didn't anticipate, from its external outputs, without shipping new code. Monitoring answers known questions; observability handles unknown ones.

Why it exists

In distributed systems the failure you hit is rarely the one you built a dashboard for. Pre-aggregated metrics say that p99 is bad, not which requests or why.

How it works

  • Metrics: cheap numeric time series for dashboards and alerts; aggregation throws detail away.
  • Logs: records of events, as detailed as you make them.
  • Traces: one request's path across services as a tree of timed spans, linked by a propagated trace ID.
  • Wide structured events ("observability 2.0"): one record per unit of work with many fields, including high-cardinality ones (user ID, build, feature flags). Metrics and traces can be derived from them; cost is controlled by sampling.
  • OpenTelemetry standardizes APIs, SDKs and the wire protocol for traces, metrics and logs, so instrumentation is vendor-neutral.

Trade-offs

Gains Costs
debug novel failures in production quickly telemetry volume, storage and cost
compare cohorts (canary vs baseline, big vs small customers) instrumentation effort; sampling can miss rare events

In practice

  • OpenTelemetry (formed 2019 from OpenTracing + OpenCensus) became a CNCF graduated project in May 2026, second only to Kubernetes in project velocity.

Pitfalls

  • Equating dashboards with observability.
  • Dropping high-cardinality fields to save cost: they're what make the "which requests?" question answerable.

Recall

These cards are also in your review deck, scheduled with spaced repetition.