Notesbatch🌱 seedling

Unix Philosophy

Also: Unix Pipes · Pipes and Filters · Do One Thing Well · Composability

Build small programs that each do one thing well and communicate through one uniform interface (streams of text), so they can be composed into jobs nobody designed them for. Distributed batch systems inherited the design.

Why it exists

Big programs that do everything are hard to change and impossible to reuse. If every tool reads and writes the same kind of stream, any tool can feed any other, and a new question is answered by recombining old tools.

How it works

  • Do one thing well: sort only sorts, uniq -c only counts adjacent duplicates.
  • Uniform interface: every program reads lines on stdin and writes lines on stdout.
  • Logic separate from wiring: programs don't know their neighbours; the shell connects them with pipes.
  • Immutable input: a pipeline never modifies its input file, so you can rerun, tweak and inspect any stage freely.
  • Scales past memory: sort spills sorted chunks to disk and merges them.
  • awk | sort | uniq -c is map, shuffle, reduce: the same shape as MapReduce, on one machine.

Trade-offs

Gains Costs
reuse and fast experimentation text parsing at every stage; no schema
each stage testable alone one machine only
no shared state between stages linear pipes; awkward for joins and multiple outputs

In practice

  • In 1986, reviewing Knuth's literate program for printing the k most frequent words in a text, McIlroy offered six piped commands (tr, tr, sort, uniq -c, sort -rn, sed).
  • Distributed systems replaced text streams with files in a distributed file system, and later with typed columnar files such as Apache Parquet.

Pitfalls

  • Parsing loosely structured text in every stage: a field containing spaces silently shifts every column.

Recall

These cards are also in your review deck, scheduled with spaced repetition.