NotesSystem
Debezium
Also: Debezium Server · Debezium Engine
The standard open-source change data capture platform: connectors that read database logs (WAL, binlog, oplog, redo log) and emit every row change as an ordered event, usually into Kafka. Its defining choice is log-based capture with a snapshot-then-stream bootstrap.
Also in: Change Data Capture & Event Sourcing › In the real world
Design choices
- Change Data Capture, log-based: Postgres through logical decoding (pgoutput) and a replication slot. MySQL and MariaDB through the binlog (ROW format, full row images). Also MongoDB, SQL Server, Oracle, Db2, Cassandra and others.
- Event envelope: before, after, source (connector, table, LSN or offset, transaction ID, commit time), op (c, u, d, r for snapshot reads, t for truncate) and ts_ms. Commit time vs processing time gives the lag.
- Keyed by primary key, so changes to one row stay ordered in one partition. Deletes are followed by a tombstone so Log Compaction can remove the row.
- Snapshots: snapshot.mode sets the initial behaviour (initial, no_data, when_needed, and more). Incremental snapshots (since 1.6) interleave primary-key chunks of 1,024 rows by default with the live log, using watermark signals, with no locks. They're triggered by writing to a signalling table.
- Transactional Outbox: the outbox event router routes outbox rows by aggregatetype to topics like outbox.event.order, keys them by aggregateid, and puts the event id in a header for deduplication.
- Runtimes: Kafka Connect connectors; Debezium Server, which writes to Kinesis, Pub/Sub, Pulsar, NATS, Redis Streams and other sinks without Kafka; and the embedded Debezium Engine Java library.
- Postgres operations: a heartbeat lets slots advance on quiet databases. Since Postgres 17, failover slots survive a primary failover.
Notable trade-offs
- At-least-once delivery: restarts redeliver recent events, so consumers must be idempotent.
- Raw CDC events expose the source schema, and schema changes flow straight to consumers unless you route through an outbox or a mapping layer.
- An unconsumed replication slot retains WAL on the source, so a stopped connector can fill the primary's disk.
Connections
- Mentioned
- Log CompactionIdempotency