Notesstorage🌱 seedling

Data Lakehouse

Also: Lakehouse · Open Table Format · Table Format

Warehouse features (ACID commits, snapshots, schema enforcement) on top of open columnar files in cheap object storage, via a table format such as Iceberg, Delta Lake or Hudi. One copy of the data, readable by many engines.

Why it exists

The two-tier design (land everything in a lake, copy a curated subset into a warehouse) pays twice for storage, adds a pipeline that breaks, and serves stale data. But a pile of Parquet files isn't a table: a half-finished write is visible, concurrent writers clobber each other, and nothing enforces a schema.

How it works

  • Data files (usually Parquet) are written first. No snapshot lists them, so they're invisible.
  • Metadata describes each snapshot: which files make up the table, with per-file statistics for pruning.
  • Commit is one atomic step: Iceberg swaps the table's metadata pointer in a catalog with compare-and-swap; Delta Lake creates the next numbered JSON file in its log with put-if-absent.
  • Conflicts are handled with Optimistic Concurrency Control: a writer whose base snapshot moved re-checks and retries.
  • Snapshots give readers a consistent view, and enable time travel and rollback until old snapshots are expired.

Trade-offs

Gains Costs
one copy, many engines, open formats you operate compaction, snapshot expiry and catalogs
cheap object storage object-store latency; small files hurt badly
ACID commits and time travel high-rate concurrent writers to one table conflict and retry

In practice

  • Iceberg (from Netflix) adds hidden partitioning; format v3 (2025) added deletion vectors, row lineage and a variant type.
  • Delta Lake writes a Parquet checkpoint of its log every 10 commits; S3 gained conditional writes only in August 2024, so Delta's documented multi-cluster S3 setup still uses a DynamoDB-backed log store.
  • Databricks agreed to buy Tabular, founded by Iceberg's creators, for over $1 billion in June 2024; AWS launched S3 Tables (managed Iceberg) that December.

Pitfalls

  • Streaming tiny commits creates thousands of small files and snapshots; schedule compaction and snapshot expiry.
  • Treating the catalog as an afterthought: it's the single point that makes commits atomic.

Recall

These cards are also in your review deck, scheduled with spaced repetition.