James Blackwood-Sewell interviews Tantivy creator Paul Masurel about keeping a search library small as more products depend on it. Tantivy handles indexing and search; the systems embedding it choose how to distribute the work. That boundary gives downstream developers room to build very different products.
Keeping it intact takes ongoing decisions:
- Prefer extension points to niche features. Maintainers sometimes reject a feature while accepting the changes needed to implement it outside the library.
- Design for developers who will customize. Masurel’s original audience needed control over tokenizers and query parsers more than a long checklist of built-in features.
- Make benchmarks shared evidence. Reproducible comparisons helped Lucene developers investigate bottlenecks, adopt ideas, and send optimization suggestions back to Tantivy. Masurel says Lucene now wins in many places.
- Budget for review. Contributor enthusiasm competes with complexity, project direction, and the maintainer’s limited attention. Saying no is part of the job.
- Protect interfaces under customer pressure. A paying customer helped fund difficult Quickwit features, but a rushed ingestion API tailored to its requirements left lasting technical debt.
A small core depends on the work around it: reviewing changes, designing escape hatches, and checking whether today’s request belongs in everyone’s dependency. Corporate backing can fund improvements while leaving less time for contributions outside the company’s agenda.
The Lucene collaboration offers a healthier model of competition, too. A benchmark that exposes weaknesses gives both projects something concrete to improve.