Notesstream🌱 seedling
Exactly-Once Semantics
Also: Exactly-Once · EOS · Effectively-Once · Exactly-Once Processing · Kafka Transactions · Idempotent Producer · Two-Phase Commit Sink
Records may be delivered and processed more than once, but the effect on state and output is as if each were processed once. It needs the progress marker (offsets, checkpoint) and the effect to commit atomically, or the effect to be idempotent.
Why it exists
At-least-once is the safe default, but duplicates corrupt counts, balances and billing. Deduplicating by hand in every consumer is error-prone.
How it works
- Within Kafka: the idempotent producer deduplicates retried appends. Transactions commit output records and consumed offsets atomically. Consumers must read with isolation.level=read_committed (the default is read_uncommitted).
- Kafka Streams: processing.guarantee=exactly_once_v2 puts offsets, state changelogs and output in one transaction per commit.
- Flink end to end: checkpoints make state exact. A two-phase-commit sink writes into a transaction, pre-commits on the checkpoint barrier and commits when the checkpoint completes. A crash aborts open transactions.
- Outside transactional systems: make the effect idempotent with a key derived from the event, or store offsets in the same database transaction as the result.
Trade-offs
| Gains | Costs |
|---|---|
| no duplicate effects, no loss | output visible only per commit or checkpoint (latency) |
| simpler consumers downstream | transactions, timeouts and fencing to operate |
| stops at the boundary of systems that can join the transaction |
In practice
- Flink's Kafka sink defaults to a 1-hour transaction timeout, but brokers cap transaction.max.timeout.ms at 15 minutes by default. Raise one or lower the other, and keep it above checkpoint plus recovery time, or data can be lost.
- Kafka producers default to idempotence and acks=all since 3.0, so duplicate appends from retries are already handled; transactions are still opt-in.
Pitfalls
- Downstream consumers left on read_uncommitted see aborted records.
- Side effects like emails and HTTP calls aren't covered. They need their own idempotency.
- "Exactly-once delivery" over a network is impossible. What's achievable is exactly-once effect.
Connections
Recall
These cards are also in your review deck, scheduled with spaced repetition.