Notesstream🌱 seedling

Delivery Semantics

Also: Delivery Guarantee · At-Least-Once · At-Most-Once · Message Delivery Semantics · Idempotent Consumer

Because any step can fail between "received" and "acknowledged", a message is delivered at-most-once (may be lost), at-least-once (may be duplicated) or effectively exactly-once (duplicates are made harmless). At-least-once with idempotent consumers is the industry default.

Why it exists

A broker can't tell a slow consumer from a dead one, and a producer whose send timed out can't tell whether it was stored. Every retry that prevents loss risks a duplicate.

How it works

  • At-most-once: ack or commit first, then process. A crash loses work. Fine for sampled metrics.
  • At-least-once: process, then ack or commit. A crash replays work, so consumers must be idempotent.
  • Exactly-once: the effect and the progress marker commit atomically (Kafka transactions, offsets stored with results), or the effect is deduplicated. See Exactly-Once Semantics.
  • Producer side: retries after a timeout can append duplicates. Kafka's idempotent producer (on by default since 3.0) numbers records per partition, so the broker drops retried duplicates.
  • Making consumers idempotent: upsert by event ID, keep a processed-ID table in the same transaction as the effect, or use naturally idempotent writes.

Trade-offs

Guarantee Gains Costs
at-most-once simplest, lowest latency data loss on failure
at-least-once no loss duplicates; dedup in every consumer
exactly-once no loss, no visible duplicates latency, coordination; stops at system boundaries

In practice

  • SQS standard queues are at-least-once with best-effort ordering; FIFO queues dedupe within a 5-minute window.
  • Kafka: acks=all plus idempotence are the producer defaults since 3.0 (fully effective from 3.0.1 / 3.2).

Pitfalls

  • Believing a broker's "exactly-once" covers your database or email provider. It ends at the broker's boundary.
  • Generating the idempotency key per attempt instead of per event makes deduplication useless.

Recall

These cards are also in your review deck, scheduled with spaced repetition.