Notesstream🌱 seedling
End-to-End Argument
Also: End-to-End Principle · End-to-End Arguments · End-to-End Correctness
A function like duplicate suppression or integrity checking can only be implemented completely by the application at the endpoints. Reliability in lower layers is a performance optimisation that makes failures rarer; it never replaces the end-to-end check.
Why it exists
Every hop can be reliable and the whole still wrong. Corruption can happen between hops (in a buffer, on a disk, in your code), and duplicates can be created above the layer that deduplicates (a user retrying).
How it works
- Careful file transfer: disks, buffers, memory, the network and host crashes can each corrupt a copy, so only reading the result back and comparing an end-to-end checksum proves it.
- Duplicate suppression: TCP removes duplicate packets within one connection, but a client retry after a lost response is a new connection. Only an operation ID generated at the origin and checked at the destination catches it.
- In streams, the request ID travels through every hop, and the final write enforces it (a unique constraint, or a dedup table updated in the same transaction).
- Lower-layer reliability still pays for itself by making end-to-end retries rare.
Trade-offs
| Gains | Costs |
|---|---|
| correctness that survives any intermediate bug | IDs and dedup state at every sink |
| freedom to use simple, unreliable middle layers | the application must reason about retries |
In practice
- Stripe idempotency keys: the client generates a unique key per operation, and the server stores it with the result and returns that result on retries.
- Kafka exactly-once covers reads and writes inside Kafka. It doesn't make a client's retried HTTP request, or a non-transactional sink, exactly once.
Pitfalls
- Believing "exactly-once" in the broker means exactly-once for the user.
- Generating the dedup ID in the middle (in the app server, per attempt) instead of once at the true origin.
Connections
- Part of
- Stream Processing
- Learn first
- Idempotency
Recall
These cards are also in your review deck, scheduled with spaced repetition.