NotesSystem

Apache Spark

Also: Spark · PySpark · Spark SQL

The dominant open-source distributed batch engine, born at UC Berkeley's AMPLab. Its defining choice is the resilient distributed dataset: intermediate data kept in memory or local disk and rebuilt from lineage on failure, not replicated.

Design choices

  • Dataflow Engine: jobs are DAGs split into stages at shuffle boundaries; narrow steps are pipelined; lost partitions are recomputed from lineage.
  • DataFrame and SQL: the Catalyst optimiser plans DataFrame and SQL queries (pushdown, join selection) and generates JVM code.
  • Join Strategies: sort-merge by default, broadcast hash below spark.sql.autoBroadcastJoinThreshold (10 MB), shuffle hash and hints.
  • Adaptive query execution (default since 3.2): re-plans at shuffle boundaries, coalescing small partitions, switching to broadcast joins and splitting skewed partitions (Data Skew: over 5× the median and 256 MB).
  • Shuffle: 200 shuffle partitions by default; push-based shuffle from LinkedIn since 3.2.
  • Releases: 4.0 (May 2025) made ANSI SQL mode the default and added VARIANT; 4.1 (December 2025) added declarative pipelines; 4.3 shipped in October 2026.

Notable trade-offs

  • Tuning memory, partitions and skew is a craft; defaults suit neither tiny nor enormous jobs.
  • Large fixed overhead: for data that fits one machine, DuckDB or Polars is usually faster.

Used by

  • Sorted 100 TB in 23 minutes on 206 EC2 machines in 2014, tying for that year's Daytona GraySort record.
  • Amazon began moving an exabyte-scale compaction workload from Spark to Ray in 2024, citing 82% better cost efficiency per GiB.