Notesbatch🌱 seedling
Unix Philosophy
Also: Unix Pipes · Pipes and Filters · Do One Thing Well · Composability
Build small programs that each do one thing well and communicate through one uniform interface (streams of text), so they can be composed into jobs nobody designed them for. Distributed batch systems inherited the design.
Why it exists
Big programs that do everything are hard to change and impossible to reuse. If every tool reads and writes the same kind of stream, any tool can feed any other, and a new question is answered by recombining old tools.
How it works
- Do one thing well:
sortonly sorts,uniq -conly counts adjacent duplicates. - Uniform interface: every program reads lines on stdin and writes lines on stdout.
- Logic separate from wiring: programs don't know their neighbours; the shell connects them with pipes.
- Immutable input: a pipeline never modifies its input file, so you can rerun, tweak and inspect any stage freely.
- Scales past memory:
sortspills sorted chunks to disk and merges them. awk | sort | uniq -cis map, shuffle, reduce: the same shape as MapReduce, on one machine.
Trade-offs
| Gains | Costs |
|---|---|
| reuse and fast experimentation | text parsing at every stage; no schema |
| each stage testable alone | one machine only |
| no shared state between stages | linear pipes; awkward for joins and multiple outputs |
In practice
- In 1986, reviewing Knuth's literate program for printing the k most frequent words in a text, McIlroy offered six piped commands (tr, tr, sort, uniq -c, sort -rn, sed).
- Distributed systems replaced text streams with files in a distributed file system, and later with typed columnar files such as Apache Parquet.
Pitfalls
- Parsing loosely structured text in every stage: a field containing spaces silently shifts every column.
Connections
- Part of
- Batch Processing
- Related
- Dataflow EngineMapReduce
- Mentioned
- Apache Parquet
Recall
These cards are also in your review deck, scheduled with spaced repetition.