Notesstorage🌱 seedling

Write-Ahead Log

Also: WAL · Redo Log · Commit Log · Journal

An append-only file where a storage engine records every change, forcing it to disk before the changed data reaches disk and before the commit is acknowledged. After a crash, replaying the log restores every acknowledged write.

Why it exists

Real data structures (B-Tree pages, a Memtable) are updated in memory and written back lazily, in many scattered places. A crash in the middle would lose or half-apply changes. One sequential append per change is cheap and is the only thing that must be durable at commit time.

How it works

  • On each write: apply the change in memory (a buffered page, or the memtable) and append a record describing it to the log.
  • At commit, fsync the log. Only after fsync returns is the write acknowledged; a plain write() only reaches the OS page cache.
  • Dirty pages reach the data files later, and never before the log records that describe them (the write-ahead rule).
  • A checkpoint flushes dirty pages, so older log segments can be recycled; recovery replays the log from the last checkpoint.
  • Group commit lets many concurrent transactions share one fsync, because fsync is the slowest step of a commit.
  • Torn pages: a page is larger than the disk's atomic write unit, so engines log a full page image on first change after a checkpoint (Postgres full_page_writes) or write pages to a doublewrite buffer first (InnoDB).

Trade-offs

Gains Costs
durability with one sequential write per commit every byte is written at least twice
data structures can be flushed lazily, in big batches fsync latency sits on the commit path
a ready-made change stream for replication log volume to store, ship and recycle

In practice

  • Postgres ships its WAL to replicas for physical replication, the Replication Log in its rawest form.
  • LSM engines like RocksDB start a new WAL file at each memtable switch and delete it once everything in it has been flushed to SSTables.

Pitfalls

  • Treating a failed fsync as retryable. Some kernels mark the dirty pages clean after an error, so a retry "succeeds" with data that never reached disk. Since 2019, Postgres crashes on fsync failure and recovers from the log instead.
  • Turning off fsync for speed silently turns committed writes into "probably committed" writes.

Recall

These cards are also in your review deck, scheduled with spaced repetition.