NotesSystem
Elasticsearch
Also: Elastic · OpenSearch · Apache Lucene · Lucene
A distributed search and analytics engine built on Apache Lucene. Its defining choice is Lucene's: an index is a set of immutable segments, each a complete [[Inverted Index]], made searchable by periodic refreshes and merged in the background.
Design choices
- Inverted Index per segment: documents buffer in memory; a refresh (every second by default, skipped for indexes not searched in 30 s unless the interval is set explicitly) writes a new segment, so search is near real-time.
- Durability: fsyncing segments (a Lucene commit) is expensive, so a transaction log covers writes in between.
- Deletes and updates: a deleted document is flagged per segment; an update is delete plus re-index; merges drop deleted documents, much like Compaction in an LSM-Tree.
- Full-Text Search: analyzers per field; BM25 scoring (k₁ 1.2, b 0.75); reciprocal rank fusion for hybrid queries.
- Approximate Nearest Neighbor: HNSW per segment (
m16,ef_construction100); float vectors default to int8 quantization (since 8.14), binary quantization (BBQ) for 384+ dimensions (9.1), and the disk-basedbbq_diskwhere licensed (9.4). - Scale-out: an index is split into primary shards with replicas; each shard is one Lucene index, capped at about 2 billion documents.
Notable trade-offs
- It is Derived Data: a second copy of the source of truth that a pipeline must keep in sync.
- Not read-your-writes by default: writes appear after the next refresh.
- Large clusters are operationally heavy. Discord moved from 200+ node clusters to 40 smaller ones grouped into cells.