Notesbatch🌱 seedling

DataFrame

Also: Data Frame · Dataframe · pandas DataFrame · Lazy DataFrame · Eager vs Lazy Evaluation · Predicate Pushdown · Projection Pushdown

A table of named, typed columns with relational operations (select, filter, join, group-by) as a library API. Eager DataFrames run each call immediately; lazy ones build a plan and optimise it like a SQL query before running anything.

Why it exists

Hand-written map and reduce functions are low-level and opaque to an optimiser. A DataFrame gives analysts a familiar table API while letting the engine see what the code means.

How it works

  • Lineage: R's data.frame → pandas (started 2008) → Spark DataFrames (2015) → Polars, DuckDB's relational API.
  • Eager (classic pandas): each operation materialises a full result, in the order written.
  • Lazy (Polars lazy API, Spark): calls build a logical plan; nothing runs until collect().
  • The optimiser applies predicate pushdown (filters move into the scan, so Parquet min/max stats skip row groups) and projection pushdown (only used columns are read), plus join reordering and common-subplan elimination, as a Declarative Query Language planner would.
  • Columnar, vectorized execution underneath (Column-Oriented Storage).

Trade-offs

Gains Costs
concise, composable, testable code eager APIs materialise every intermediate
lazy plans optimised like SQL lazy errors surface at collect(), far from the line at fault
one API from laptop to cluster per-row Python UDFs bypass the optimiser and are slow

In practice

  • In the Spark SQL paper's aggregation benchmark, DataFrame code beat hand-written RDD code 2× in Scala and 12× in Python.
  • Single-node engines go far: DuckDB ran TPC-H at scale factor 10,000 (about 10 TB) on a laptop in 4.2 hours in 2025.

Pitfalls

  • collect() or toPandas() on a big Spark DataFrame pulls everything onto the driver.
  • Row-by-row apply() with a Python function: an eager loop the engine can't optimise.

Recall

These cards are also in your review deck, scheduled with spaced repetition.