Chunking Inside Manticore Search — Dmitrii Kuzmenkov

Start with the failure this feature exists to fix. You have an embedding model with a 512-token input window — roughly 380 English words. You insert a 4,000-word document. The insert succeeds. Search works. Everything looks fine. The model read the first few hundred words and threw the other 3,600 away, and nothing anywhere told you. Anything past that point can never be retrieved.

Until now the answer was to split documents yourself, embed each piece, and work out how to fold the results back into documents. Manticore moved that into the table definition — one option on a vector column:

CREATE TABLE docs (
  title text, content text,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,content'
    chunk_strategy='sentence' max_tokens='256' overlap_tokens='32'
);

No ingest pipeline, no splitter library, no second table for chunks, no GROUP BY.

The reframing, not the plumbing

The interesting part is what a match means. With one vector per document, search asks “is this document, as a whole, similar to the query?” A single relevant paragraph gets diluted by everything around it. With one vector per chunk, search asks “does this document contain something similar?”

Consequences worth knowing: a document matches if any of its vectors is close, it is returned exactly once, knn_dist() reports the distance to its closest chunk, k counts documents rather than chunks, and queries are never chunked — a fifteen-word question is short enough to embed whole.

Five strategies

• truncate — the old default. Embed what fits the window, silently drop the rest. Right choice for titles, tags, chat messages, log lines.
• mean — split, embed everything, average into one vector. Same storage cost as truncate, nothing discarded. Fits long-but-single-topic documents, or a corpus where RAM is the binding constraint.
• fixed — cut every N tokens. Predictable, no structure needed. Use it for OCR output, scraped HTML that lost its paragraphs, punctuation-free transcripts.
• recursive — same budget, but each cut snaps back to the nearest natural boundary: paragraph, then line, then sentence, then space. Their best scorer on buried content, and the right default for documentation, wikis and READMEs.
• sentence — sentence boundaries, packed greedily. Pick it when a fragment changes the meaning (legal text, support tickets, transcripts) or when the chunks get pasted into an LLM prompt.

The numbers

Their benchmark is their own 189-page, 298k-word English manual, with queries generated mechanically from section headings rather than hand-picked. For headings buried past the model’s window (419 queries):

About 2.5x the RAM and 4x the ingest time for a working search instead of a coin flip. The ingest cost is embedding cost, not splitting cost — which matters if you are paying an API per token. For content near the top of a page, truncate is still the most precise at rank 1 (65.9% vs 58.0%), because a whole-document vector carries the page’s overall topic; by rank 5 the difference is gone.

The part that contradicts the usual advice

A second sweep over chunk size and overlap found that smaller chunks won consistently — at 128 tokens, recall@5 was 85.2% and MRR 0.718, against 80.2% and 0.655 at 512 tokens. And overlap did nothing measurable for quality while adding 41% more vectors. The spread across overlap settings sat within the noise of a 419-query set; the cost did not.

So their default is 128-256 tokens with no overlap, adding overlap only if you can measure it helping — the opposite of the “always use 10-20% overlap” guidance in most RAG write-ups. They cite Chroma’s own chunking evaluation agreeing, and are upfront that this is one corpus, one model, one query style. Overlap earns its place where a fact routinely straddles a boundary — long unbroken narrative, unstructured transcripts — and recursive already does much of that work by snapping cuts to real boundaries.

Gotchas

• Remote models (OpenAI, Voyage, Jina) have no local tokenizer, so Manticore falls back to 3 bytes per token. max_tokens='N' becomes an N x 3-byte window, giving chunks about a quarter smaller than you asked for. Bump it ~30%, or use a local model for real token boundaries.
• max_chunks silently discards text — overflow is merged into the last kept chunk and then truncated to the model window, with no warning. It is a guard rail for outliers, not a memory-saving strategy.
• Multi-vector columns cannot be added with ALTER yet; there is no backfill. Recreate the table and reindex. Single-vector mean works fine with ALTER ... REBUILD EMBEDDINGS.
• No per-chunk prefixing. Fields in from='title,body' are joined before chunking, so the title lands in the first chunk only. Every later chunk has no idea what document it came from — the exact gap Anthropic’s contextual retrieval work measured at a 35% cut in top-20 retrieval failures. You have to build that context into the stored text yourself.
• Late chunking (embed the long document first, then pool into chunks) is not supported.
• Bad combinations fail at CREATE TABLE time with a real error message, which is worth more than it sounds: sentence on a float_vector, overlap_tokens without max_tokens, and an unknown strategy name are all rejected up front.

The wider point is that in-engine chunking is still rare. Most vector databases will happily run the embedding model for you and quietly truncate your 4,000-word document; the splitting stays your problem, in a library that has no idea which tokenizer the model uses. Elasticsearch’s semantic_text is the closest thing to this design. The rest make you assemble it from pipeline stages, nested fields, or a second table you have to join back.

Honest caveat: this is a vendor benchmark on a vendor’s own documentation, with queries that look like headings. The direction of the result is unsurprising and well-established; the specific numbers are the vendor’s.