Notesstorage🌱 seedling
Write-Ahead Log
Also: WAL · Redo Log · Commit Log · Journal
An append-only file where a storage engine records every change, forcing it to disk before the changed data reaches disk and before the commit is acknowledged. After a crash, replaying the log restores every acknowledged write.
Why it exists
Real data structures (B-Tree pages, a Memtable) are updated in memory and written back lazily, in many scattered places. A crash in the middle would lose or half-apply changes. One sequential append per change is cheap and is the only thing that must be durable at commit time.
How it works
- On each write: apply the change in memory (a buffered page, or the memtable) and append a record describing it to the log.
- At commit,
fsyncthe log. Only after fsync returns is the write acknowledged; a plainwrite()only reaches the OS page cache. - Dirty pages reach the data files later, and never before the log records that describe them (the write-ahead rule).
- A checkpoint flushes dirty pages, so older log segments can be recycled; recovery replays the log from the last checkpoint.
- Group commit lets many concurrent transactions share one fsync, because fsync is the slowest step of a commit.
- Torn pages: a page is larger than the disk's atomic write unit, so engines log a full page image on first change after a checkpoint (Postgres
full_page_writes) or write pages to a doublewrite buffer first (InnoDB).
Trade-offs
| Gains | Costs |
|---|---|
| durability with one sequential write per commit | every byte is written at least twice |
| data structures can be flushed lazily, in big batches | fsync latency sits on the commit path |
| a ready-made change stream for replication | log volume to store, ship and recycle |
In practice
- Postgres ships its WAL to replicas for physical replication, the Replication Log in its rawest form.
- LSM engines like RocksDB start a new WAL file at each memtable switch and delete it once everything in it has been flushed to SSTables.
Pitfalls
- Treating a failed fsync as retryable. Some kernels mark the dirty pages clean after an error, so a retry "succeeds" with data that never reached disk. Since 2019, Postgres crashes on fsync failure and recovers from the log instead.
- Turning off fsync for speed silently turns committed writes into "probably committed" writes.
Connections
- Part of
- Storage
- Builds on this
- MemtableReplication Log
- In real systems
- RocksDBPostgreSQLMySQL
- Mentioned
- SSTable
Recall
These cards are also in your review deck, scheduled with spaced repetition.