NotesSystem

Elasticsearch

Also: Elastic · OpenSearch · Apache Lucene · Lucene

A distributed search and analytics engine built on Apache Lucene. Its defining choice is Lucene's: an index is a set of immutable segments, each a complete [[Inverted Index]], made searchable by periodic refreshes and merged in the background.

Design choices

  • Inverted Index per segment: documents buffer in memory; a refresh (every second by default, skipped for indexes not searched in 30 s unless the interval is set explicitly) writes a new segment, so search is near real-time.
  • Durability: fsyncing segments (a Lucene commit) is expensive, so a transaction log covers writes in between.
  • Deletes and updates: a deleted document is flagged per segment; an update is delete plus re-index; merges drop deleted documents, much like Compaction in an LSM-Tree.
  • Full-Text Search: analyzers per field; BM25 scoring (k₁ 1.2, b 0.75); reciprocal rank fusion for hybrid queries.
  • Approximate Nearest Neighbor: HNSW per segment (m 16, ef_construction 100); float vectors default to int8 quantization (since 8.14), binary quantization (BBQ) for 384+ dimensions (9.1), and the disk-based bbq_disk where licensed (9.4).
  • Scale-out: an index is split into primary shards with replicas; each shard is one Lucene index, capped at about 2 billion documents.

Notable trade-offs

  • It is Derived Data: a second copy of the source of truth that a pipeline must keep in sync.
  • Not read-your-writes by default: writes appear after the next refresh.
  • Large clusters are operationally heavy. Discord moved from 200+ node clusters to 40 smaller ones grouped into cells.