Notesbatch🌱 seedling
DataFrame
Also: Data Frame · Dataframe · pandas DataFrame · Lazy DataFrame · Eager vs Lazy Evaluation · Predicate Pushdown · Projection Pushdown
A table of named, typed columns with relational operations (select, filter, join, group-by) as a library API. Eager DataFrames run each call immediately; lazy ones build a plan and optimise it like a SQL query before running anything.
Why it exists
Hand-written map and reduce functions are low-level and opaque to an optimiser. A DataFrame gives analysts a familiar table API while letting the engine see what the code means.
How it works
- Lineage: R's data.frame → pandas (started 2008) → Spark DataFrames (2015) → Polars, DuckDB's relational API.
- Eager (classic pandas): each operation materialises a full result, in the order written.
- Lazy (Polars lazy API, Spark): calls build a logical plan; nothing runs until collect().
- The optimiser applies predicate pushdown (filters move into the scan, so Parquet min/max stats skip row groups) and projection pushdown (only used columns are read), plus join reordering and common-subplan elimination, as a Declarative Query Language planner would.
- Columnar, vectorized execution underneath (Column-Oriented Storage).
Trade-offs
| Gains | Costs |
|---|---|
| concise, composable, testable code | eager APIs materialise every intermediate |
| lazy plans optimised like SQL | lazy errors surface at collect(), far from the line at fault |
| one API from laptop to cluster | per-row Python UDFs bypass the optimiser and are slow |
In practice
- In the Spark SQL paper's aggregation benchmark, DataFrame code beat hand-written RDD code 2× in Scala and 12× in Python.
- Single-node engines go far: DuckDB ran TPC-H at scale factor 10,000 (about 10 TB) on a laptop in 4.2 hours in 2025.
Pitfalls
- collect() or toPandas() on a big Spark DataFrame pulls everything onto the driver.
- Row-by-row apply() with a Python function: an eager loop the engine can't optimise.
Connections
- Part of
- Batch Processing
- Learn first
- Relational Model
- In real systems
- Apache Spark
Recall
These cards are also in your review deck, scheduled with spaced repetition.