Notesfoundations🌱 seedling
Observability
Also: O11y · Distributed Tracing · Telemetry · Wide Events
Observability is the ability to understand what a system is doing, and answer questions you didn't anticipate, from its external outputs, without shipping new code. Monitoring answers known questions; observability handles unknown ones.
Why it exists
In distributed systems the failure you hit is rarely the one you built a dashboard for. Pre-aggregated metrics say that p99 is bad, not which requests or why.
How it works
- Metrics: cheap numeric time series for dashboards and alerts; aggregation throws detail away.
- Logs: records of events, as detailed as you make them.
- Traces: one request's path across services as a tree of timed spans, linked by a propagated trace ID.
- Wide structured events ("observability 2.0"): one record per unit of work with many fields, including high-cardinality ones (user ID, build, feature flags). Metrics and traces can be derived from them; cost is controlled by sampling.
- OpenTelemetry standardizes APIs, SDKs and the wire protocol for traces, metrics and logs, so instrumentation is vendor-neutral.
Trade-offs
| Gains | Costs |
|---|---|
| debug novel failures in production quickly | telemetry volume, storage and cost |
| compare cohorts (canary vs baseline, big vs small customers) | instrumentation effort; sampling can miss rare events |
In practice
- OpenTelemetry (formed 2019 from OpenTracing + OpenCensus) became a CNCF graduated project in May 2026, second only to Kubernetes in project velocity.
Pitfalls
- Equating dashboards with observability.
- Dropping high-cardinality fields to save cost: they're what make the "which requests?" question answerable.
Connections
- Part of
- Operability
Recall
These cards are also in your review deck, scheduled with spaced repetition.