NotesSystem
Apache Parquet
Also: Parquet · Parquet File Format
An open columnar file format, announced by Twitter and Cloudera in 2013, that nearly every analytics engine reads and writes. Its defining choice is a footer with the schema, chunk offsets and statistics, written after the data, so readers fetch only what a query needs.
Design choices
- Column-Oriented Storage per row group: rows split into row groups (parquet-java targets 128 MB; the spec suggests 512 MB–1 GB), each holding one contiguous column chunk per column.
- Pages are the unit of encoding and compression (1 MB by default in parquet-java); a chunk has an optional dictionary page, then data pages.
- Columnar Compression: dictionary encoding (RLE/bit-packed codes) by default, with fallback to PLAIN when the dictionary grows too big; also DELTA_BINARY_PACKED, BYTE_STREAM_SPLIT and others, then Snappy, Zstandard, LZ4 or gzip.
- Nested data is shredded into columns with repetition and definition levels, the encoding from Google's Dremel.
- Skipping: min/max statistics per chunk, an optional page index with per-page min/max, and optional per-chunk Bloom Filters for high-cardinality columns.
- Footer last: the file ends with the metadata, its 4-byte length and the magic PAR1, so writers stream in one pass and readers start from the end.
Notable trade-offs
- Files are immutable: no in-place updates. Table formats (Data Lakehouse) add commits, deletes and snapshots on top.
- Many small files mean many footers and little skipping; files need compacting into large row groups.
- Optimised for storage, not computation: engines decode it into an in-memory format such as Apache Arrow.
Used by
- Apache Spark, Trino, DuckDB, Polars, Snowflake, BigQuery and most other engines; the default data-file format of Iceberg and Delta Lake.
Connections
- Builds on this
- Data Lakehouse
- Related
- Columnar Compression
- Mentioned
- Apache Spark