Rag

Isaac Flath walks Hamel Husain through a 42-minute hands-on evaluation of OCR and document-question-answering agents. His key move: inspect the document, extracted text, retrieval, citations, and agent trace together. Why a wrong answer may not be a model failure A misread digit in a financial table can look like an LLM hallucination; better prompting cannot correct corrupted input. Documents mix scanned images, text layers, tables, watermarks, and handwriting. Medical, legal, and financial workflows can turn small extraction errors into consequential answers. Judge errors by the task: a shifted underline may be cosmetic; a missing inspection score or draft watermark may change the meaning. Inspect real documents, not just benchmark scores On an ordinary food-inspection form, a heading spans rows inconsistently and an OCR model drops a point value. Even the presenters debate the correct grouping. Compare the PDF, raw OCR, and rendered output side by side. Attach each observation to both a page bounding box and the corresponding extracted-text span. A different model invents checkboxes, converting ambiguous fields into binary ones; both OCR systems miss an “example” watermark in a medical record. A missed “draft,” “confidential,” or “void” stamp could be much worse. Flath built his annotation UI incrementally, starting in about an hour and adding rendered views, boxes, and hover-linked text whenever manual review became painful. Trace an agent failure backwards Check the final answer and whether its citation actually supports it. Inspect the agent trace: was the wrong number already in its context? Check retrieval: did it fetch the wrong page or the right page with wrong text? Compare extracted text with the original PDF. If OCR broke it, test and fix that layer, not the prompt. Add a cheap targeted check when possible, such as verifying table entries add up to the stated total. Evals across the pipeline Test OCR, retrieval, and the agent loop separately; swapping an agent harness will not repair a faulty source representation. A citation’s presence is weaker than checking whether the cited image/text supports the answer. Visual, page-level citations make human verification practical. Tool-using agents can crop and re-OCR pages until they find an answer, but sometimes take 45 minutes and spend $20, even when the answer is absent. “Blank” versus “zero” in a loan estimate, or a faint signature on a contract, requires domain judgment; not every discrepancy admits an automatic yes/no label. “Maybe 80% of my failures are actually just because my OCR model failed, in which case prompt tuning is just a waste of my time right now.” — Isaac Flath (hypothetical example, not a measured result) ...

Cole Medin (16 min) is a year into his personal AI second brain — an agent that builds a knowledge base as he works — and this one answers the question he says he keeps getting: how do you turn a second brain into a team brain? ...

Nandan Thakur — the University of Waterloo PhD behind BEIR, MIRACL, FreshStack and TREC-RAG, now a postdoc at Microsoft Research India — on Hamel Husain’s channel, ~51 minutes on the three things you need if you’re building a search agent: a benchmark you can actually reproduce, synthetic training data you can afford, and a way to see what the agent is doing under the hood. He’s upfront that the field is young: “not many things are formalized yet.” ...

Pete Johnson — MongoDB’s field CTO of AI and a 30-year database veteran — makes the case across ~95 minutes on The Cognitive Revolution that the interesting frontier in AI has moved back into database territory. His thesis, stated once: agent performance, and especially cost-adjusted agent performance, depends on retrieval — what you choose to put in front of the model, in what order. The thesis Not the model, not the context window, not the prompt — retrieval is what decides whether an agent is good Everything else in the conversation (database history, the Voyage acquisition, vector search) is downstream of that one claim A history of constraints 1970: SQL is born (E.F. Codd, IBM) — storage was the scarce resource, so normalization (store nothing twice) was the right design 2007: MongoDB’s first commit, the year the iPhone ships — after 47 years of Moore’s Law, time became scarce, so denormalize: one JSON document, one disk read instead of three “The problem has faded, but the solution persists” — Johnson agrees, and flags the education system’s “thou shalt always normalize” bias MongoDB’s AI path — it started with keyword search 2020: customers stood up their own Lucene servers for keyword search → MongoDB shipped Atlas Search (lexical, managed) A vector is just an array of floats → to a document DB, that’s just another attribute, so vector search was cheap to add Three levers: pre-filter (metadata) + lexical + vector = hybrid search in one query (rank fusion / score fusion, one API call) 2025: the Voyage acquisition — and the conversation pivots from “database features” to “embeddings actually matter” Embeddings are not commoditized “Most people think embedding models are commoditized — that is not true” Hugging Face’s Rtech benchmark: up to a 14% quality gap vs. the default picks 14% is the difference between a hallucination and a correct answer A reranker on top adds another 5–10% ($re-rank, one-call) Anthropic — no embedding model of its own — recommends Voyage Three Voyage features that remove plumbing Contextualized chunking: send the chunk plus its surrounding context, get one vector back — better retrieval at smaller chunks, inverting the normal tradeoff Matryoshka reasoning: dimensions nest like Russian dolls — embed at 1024, lop off the last 512 to test, no re-embedding your corpus Shared embedding spaces: four sizes of one model share an embedding space; a free open-weight “nano” can run queries locally to kill token cost in dev The memory problem, compressed 2022: query → context window → answer. 2023: the knowledge cutoff + proprietary data → RAG. 2025: tools/MCP + looping → the memory problem Early answer: short-term memory = cram the session; long-term = cram the last three days Two failures: token maxing (Uber burned its entire 2026 budget in 13 weeks) and lost-in-the-middle (the first and last ~7K tokens are what matter; the middle muddies the answer) The fix is selection, not stuffing Stop asking “how do I cram a million tokens in” — ask “how do I choose the right 200K for this loop” Taxonomic memory: a hundred company-specific terms exist, but only five are relevant to this loop — pick those five, re-pick next loop Two responsibilities now: query with a token budget, and write the answer back so the system curates and stores it Write, change, recall, forget Memories have a half-life — recent matters more — and forgetting is the hardest part Nathan’s own memory system (monthly logs → yearly summaries → entity wiki) hits both pain points: the DRY violation and the model keeping a dead project open for months Guidance: a good embedder + reranker makes the forget step workable; graph structure for the top 2–6 levels, vector search in the leaf; don’t run multiple LLM passes to shrink the corpus — that’s just more tokens Memory done well: ElevenLabs’ micro-agents, one per customer Build vs. buy, three camps Camp one: “I bought one tool, I’m done.” Camp two: POC purgatory — usually the wrong problem. Camp three: optimizing sophisticated memory Problem selection: top 10–15 problems, which have good data, which already have metrics — else you can’t tell if AI helped “Bad data quality and bad security posture don’t get solved by AI — they get amplified” Lines of code is a terrible metric; idea-to-production is the one that matters The world outside the US Seven countries, ~100 customers this year — and the two most sophisticated were in Mexico City and São Paulo, both assuming US competitors were ahead Nearly every country has a hyperscaler data center now — the geographic barriers that kept US companies ahead have eroded “We’ve been building databases for 60 years. We’ve been building agents for about 18 months… there’s no LAMP stack for agents yet — no React and Angular, no established right answer an enterprise can confidently buy.” ...

Radu, a search engineer at Vespa, on how to actually choose and tune embedding models for search and RAG — hosted by Hamel Husain. ~22 minutes. Why not just take the leaderboard Most people pick embeddings one of two ways: the top of a benchmark leaderboard, or the model from whatever provider they already use — leaving a lot on the table Cost, latency, robustness to future model changes, and your actual use case aren’t reflected in a single ranking MTEB is a starting point, not an answer: find the subtask closest to your use case, restrict by model size, and watch for small models that punch above their weight (this talk covers dense embedders — one vector per chunk) What MTEB doesn’t tell you Quantization depends on where you run: on CPU, int8 keeps most of the precision at a fraction of the cost and runs much faster; on GPU, an int8 model runs at its native rate and is actually slower — run FP16 instead, basically the same result for significantly less Vector precision: storing vectors as bf16 instead of float32 is a no-brainer (no measurable loss); bit vectors (packed to int8) do cost quality — but in hybrid search the gap shrinks and they become viable Matryoshka dimensions: some models are trained so the earliest dimensions matter most, so you can just cut the vector — 2048 → 1024 dims cost nothing, 512 a little, 256 more; and 2048-dim bit vectors run a quarter the size of bf16 at 512 dims Query latency comes from two places: how fast the embedder turns the query into a vector, and the distance math — normalized vectors let you use dot product instead of cosine, bit vectors unlock Hamming distance (fastest, and CPU-optimized in modern search engines) Constraints MTEB never covers: multilingual needs, long-context support Tune on your own data with Vespa Embed Open-source fine-tuning tool (UI over sentence-transformers, models from Hugging Face): feed pairs (query → document) or triplets when you have negatives; it auto-splits a validation set MNR (multiple negatives ranking) treats other documents in the batch as negatives; symmetric and JSD variants check whether a batch-picked document is actually a negative Start with defaults; if you have labeled negatives, try triplets — his e-commerce results were similar either way, but real hard negatives (from search logs: positives rank on top, negatives below) are worth mining No clean labels? Use an LLM as a judge: have it rank query→document pairs on a 0/1/2 scale, give it examples (the most important part), and run micro-batches of 8-10 docs so it doesn’t assume listwise context Fine-tune embeddings far more readily than LLMs — it’s a constrained problem: cheap, quick, and consistently a big NDCG jump The cheap-embedder escape hatch: re-rankers Two-phase search: an intentionally “great but not greatest” cheap embedder does first-phase ranking over millions of docs, then a more expensive re-ranker (float vectors, cross-encoders, late interaction) scores only the top N Store cheap bit vectors (with HNSW) in memory for the first phase and keep float vectors on disk for the re-rank — memory is expensive, disks aren’t Late-interaction re-ranking (MaxSim) is literally: per-token dot products, keep the max per patch, sum across the document Do’s and don’ts from real experiments Don’t train on title + description as a proxy for queries — proxies lie; people’s real queries look nothing like product copy. Train on real queries and relevance judgments NDCG has a blind spot: change your relevance function and newly-surfaced documents score zero until they’re rated — if you don’t re-rate everything in the metric, your NDCG looks like it never improves Pick the metric by the problem: e-commerce ranking → NDCG/ERR (top results matter); RAG → precision matters more, because junk retrieved = hallucination, and recall is capped by context size Don’t over-optimize: with ~2M documents most of these knobs don’t matter yet — start with defaults that lose little, then measure “The embedding is a much more constrained problem — and it’s a lot easier to fine-tune. I always find a really good benefit from doing it. And it’s not that complicated.” — Hamel Husain ...

Ofer Mendelevitch (Vectara; author of Hands-On RAG for Production and, with Jay Alammar, Hands-On Large Language Models) interviewed by Angelina on TwoSetAI — 70 minutes on production RAG and surviving the AI shift. RAG isn’t dead — it just got a loop RAG = retrieval + augmented generation, and retrieval stays essential even with agents; the “RAG is dead” claims every two months are mostly people marketing something new Classic RAG is one-shot: query → top chunks → prompt → answer. Agents add a loop: the LLM plans, calls tools (often the RAG pipeline itself) repeatedly, and synthesizes “Talk to my PDF” demos are not production: millions of documents in every format change the problem completely The pipeline, from ingest to answer Ingest: extract text → chunk → embed → vector store (store the text and page markers too, not just vectors — citations need to point at exact pages) Hybrid search (semantic + BM25/TF-IDF) for what semantic search misses: numbers, product codes, exact strings — “if the source says 90% and your output says 85%, vector search won’t catch it” Query side: top-5/10 results → optional reranking → prompt → grounded answer Tables, images, and the red button problem Tables are first-class citizens: chunked tables lose their column names (common in medical journals) — store whole tables, retrieve them whole Images: store and return as images, don’t flatten to a description Video: transcription alone loses meaning — “in order to avoid catastrophe, never press this button” means nothing without the visual. Today’s fix: VLM descriptions of short clips correlated with the transcript. Dedicated video embedding models exist but aren’t production-ready yet When to bother with knowledge graphs Multi-hop questions (“what else did the director of Inception direct?”) defeat semantic search But graphs are expensive to build and maintain — worth it only for high-stakes use cases where a significant share of queries actually need the relationships Eval: the hard part is the data, not the metric Retrieval eval (did you fetch the right chunks?) needs query→gold-chunk datasets that are brutal to build — and documents keep changing Generation eval compares against curated golden responses Reference-free eval (Jimmy Lin’s Waterloo lab + Vectara): LLM-as-judge scorers like UMBRELLA (0–3 chunk relevance) validated against human correlation — no gold labels required Build, buy, or rent Build with LangChain/LlamaIndex only if it’s your business and you have the team; complexity compounds (multimodal, graphs, maintenance) RAG/agent-as-a-service (e.g. Vectara) outsources the upkeep; vertical tools are fine when they cover your use case — but watch missing features and data-residency constraints DevRel as a growth engine Two jobs: teach developers how to use the product, and carry feedback back to the company PLG over expensive sales teams: self-serve product, events, hackathons, real blog posts — “make it your own voice, don’t produce AI slop” Measure directionally, don’t over-engineer attribution: five customers means it’s not working, ten thousand means it is, a thousand is unknowable — same problem founders face reading PMF The skill that survives Engineers and data scientists become directors, not actors: agents write the code; the remaining critical challenge is deciding what to build and steering where agents are weak (architecture, non-obvious trade-offs) To the high schoolers who feared they made “the most incredibly stupidest mistake” by majoring in CS: graduate with the capability of today’s mid/senior engineer — use college to learn how to wield the AI tools Fundamentals still matter; hiring will have to change — “write Fibonacci in five lines of Python is worthless” — expect AI-augmented interviews His real worry is societal, not technical: how governments and finance distribute the gains “We’re going to end up in engineering and data science being directors as opposed to actors. The coding agents will write the code.” ...

RAG — retrieval-augmented generation — is the technique that lets an AI answer questions about your own documents: it searches them first, then reads the best matches to compose an answer. Most teams build this the hard way, jumping straight to vector databases and reranking pipelines. Rafael Pierre’s essay argues that’s usually backwards: a plain keyword search handles a surprising share of real queries, and you should only climb the complexity ladder when you have data proving you need to. ...

Hamel Husain hosts Marek Galovic, CEO and co-founder of Top-K (ex-Pinecone data plane lead, ex-Shopify), on scaling multi-vector / late-interaction retrieval. ~24 minutes. Why single-vector embeddings fail agents Pooling is a lossy summary — it captures high-level semantics but drops the low-level detail precise queries need Agents issue many specific parallel queries; single-vector retrieval returns the same documents for all of them Agentic retrieval is sequential — noisy retrievals compound errors over multiple hops DeepMind’s limit paper: single vectors can’t capture arbitrary relevance matrices, even with infinite dimensions (embeddings are low-rank) Multi-vector = chop off the pooling layer Keep one embedding per token; score every query token against every document token (max-sim), then aggregate Preserves low-level detail; much better on out-of-domain and long-context retrieval Cost: 10-100x more storage, ~3 orders of magnitude more flops per score Existing workarounds (ColBERT-style compression into IVFPQ indexes) make updates and filtering hard in production Sparse multi-vector encoding (Top-K’s approach) Random projections map token embeddings into tens-of-thousands-dimensional space, then sparsify (keep top-k per token) Aggregating token-level sparse vectors into one document/query vector makes the dot product approximate max-sim Retrieval becomes inverted posting lists like BM25 — cost scales with non-zeros, not ambient dimension Two-stage: prune a billion docs to a few hundred candidates, then re-rank with exact max-sim (1-2 bit quantization, custom kernels, tens of thousands of docs/sec/core) Production numbers Sub-50ms P99 at billion scale; hundreds of QPS; 70MB/s writes with no query-latency impact Object storage as the durable layer + stateless compute; separate read/write pools Quality: a 100M-param multi-vector model outperformed an 8B dense model by ~40% on some video-doc retrieval; on BrowseComp an off-the-shelf 120B open model + multi-vector matched a proprietary GPT-5 setup; OfficeQA Pro went 18% @ $6/query → 42% @ $0.50/query Practical tuning advice Start with evals on your own private data, then hill-climb Dimensions are usually 128; you can prune tokens and quantize without hurting recall Relevance tuning (content score × user signals like distance/popularity) is underrated — e-commerce does it better than RAG teams “If you just vibe it, you can get better vibes from the system, but that’s not systematic. You need evals to know where you stand — and know if you’re improving or regressing.” ...