Notesstream🌱 seedling

Log Compaction

Also: Compacted Topic · Tombstone · cleanup.policy=compact · Key-Based Retention

Retention by key instead of by age: the broker keeps at least the latest record for every key and discards older ones, so replaying the topic from offset 0 always rebuilds the current table. A null-valued tombstone deletes a key.

Why it exists

Time-based retention suits event streams but breaks streams of state changes: delete by age and you lose keys that simply haven't changed lately. Keeping everything forever wastes space on superseded values.

How it works

  • With cleanup.policy=compact, a background cleaner rewrites old segments, keeping only the newest record per key. The active segment is never cleaned.
  • Survivors keep their original offsets (gaps appear) and their relative order.
  • A record with a null value is a tombstone. It is kept for delete.retention.ms (24 h by default) so consumers see the delete, then removed.
  • A segment becomes eligible once its dirty share passes min.cleanable.dirty.ratio (0.5 by default).
  • The idea is the same as LSM Compaction, applied to a log: newest version wins, tombstones expire.

Trade-offs

Gains Costs
bounded size for unbounded history of state no history of intermediate values
any consumer can bootstrap the full state cleaner I/O; space reclaimed lazily
deletes propagate as tombstones consumers lagging more than delete.retention.ms may miss deletes

In practice

  • Kafka stores consumer offsets in a compacted topic. Kafka Streams backs state stores with compacted changelogs, and Kafka Connect keeps configs and offsets in compacted topics.
  • CDC: a compacted topic of row changes keyed by primary key is a replayable copy of the table.

Pitfalls

  • Compaction is not immediate: old values linger until the cleaner runs, so don't rely on it alone for Right to Erasure.
  • Records without keys can't be compacted.

Recall

These cards are also in your review deck, scheduled with spaced repetition.