Hamel Husain hosts Marek Galovic, CEO and co-founder of Top-K (ex-Pinecone data plane lead, ex-Shopify), on scaling multi-vector / late-interaction retrieval. ~24 minutes.

Why single-vector embeddings fail agents

  • Pooling is a lossy summary — it captures high-level semantics but drops the low-level detail precise queries need
  • Agents issue many specific parallel queries; single-vector retrieval returns the same documents for all of them
  • Agentic retrieval is sequential — noisy retrievals compound errors over multiple hops
  • DeepMind’s limit paper: single vectors can’t capture arbitrary relevance matrices, even with infinite dimensions (embeddings are low-rank)

Multi-vector = chop off the pooling layer

  • Keep one embedding per token; score every query token against every document token (max-sim), then aggregate
  • Preserves low-level detail; much better on out-of-domain and long-context retrieval
  • Cost: 10-100x more storage, ~3 orders of magnitude more flops per score
  • Existing workarounds (ColBERT-style compression into IVFPQ indexes) make updates and filtering hard in production

Sparse multi-vector encoding (Top-K’s approach)

  • Random projections map token embeddings into tens-of-thousands-dimensional space, then sparsify (keep top-k per token)
  • Aggregating token-level sparse vectors into one document/query vector makes the dot product approximate max-sim
  • Retrieval becomes inverted posting lists like BM25 — cost scales with non-zeros, not ambient dimension
  • Two-stage: prune a billion docs to a few hundred candidates, then re-rank with exact max-sim (1-2 bit quantization, custom kernels, tens of thousands of docs/sec/core)

Production numbers

  • Sub-50ms P99 at billion scale; hundreds of QPS; 70MB/s writes with no query-latency impact
  • Object storage as the durable layer + stateless compute; separate read/write pools
  • Quality: a 100M-param multi-vector model outperformed an 8B dense model by ~40% on some video-doc retrieval; on BrowseComp an off-the-shelf 120B open model + multi-vector matched a proprietary GPT-5 setup; OfficeQA Pro went 18% @ $6/query → 42% @ $0.50/query

Practical tuning advice

  • Start with evals on your own private data, then hill-climb
  • Dimensions are usually 128; you can prune tokens and quantize without hurting recall
  • Relevance tuning (content score × user signals like distance/popularity) is underrated — e-commerce does it better than RAG teams

“If you just vibe it, you can get better vibes from the system, but that’s not systematic. You need evals to know where you stand — and know if you’re improving or regressing.”