Nandan Thakur — the University of Waterloo PhD behind BEIR, MIRACL, FreshStack and TREC-RAG, now a postdoc at Microsoft Research India — on Hamel Husain’s channel, ~51 minutes on the three things you need if you’re building a search agent: a benchmark you can actually reproduce, synthetic training data you can afford, and a way to see what the agent is doing under the hood. He’s upfront that the field is young: “not many things are formalized yet.”
From RAG to agentic search
- RAG retrieved once, then generated. Query → retriever → top-K context → LLM answer, in one pass
- Agentic search flips the driver. The LLM goes first: it decides whether to search, issues queries to a retrieval tool, reads results, refines, and iterates until it answers. Retrieval and reasoning are interleaved
- Search tends to be the weakest point in most AI products, and the literature is already exploding with search-agent papers and benchmarks
Evaluating search agents is not RAG evaluation
- Existing retrieval/QA benchmarks (HotpotQA, Natural Questions, BEIR) are simple factoid queries — not hard enough to test an agent that reasons across multiple searches
- Riddle-style questions (BrowseComp, from OpenAI) are the opposite shape: easy to verify, painful to search — e.g. title of a pre-June-2023 publication co-authored by an assistant professor in West Bengal and a PhD holder
- Solving them is a funnel search: 100+ papers matching criterion one → ~20 after co-author count → one after the last constraint
- BrowseComp queries were kept only if a human with search tools couldn’t solve them in 10 minutes, and GPT-4 couldn’t either
- Why reported BrowseComp numbers aren’t comparable: papers rarely state their search tool (which swings EM a lot), nobody audits whether one model got a better retriever than another, and EM-only scores hide how much the retrieval tool contributed
BrowseComp-Plus: turning QA into an open-retrieval problem
- Take BrowseComp’s question-answer pairs and convert them to a retrieval evaluation setup: OpenAI o3 gathered candidate gold web pages, then human annotators judged which documents answer which part of each question
- Result: a ~100k-document corpus (positives + hard negatives, so retrieval is genuinely hard), plus per-query relevance judgments
- Metrics: EM accuracy for answers; search-call count and positive-passage recall for the retrieval side
The retriever limits accuracy more than the model
- Both the LLM and the retriever contribute, but the retriever is the bigger lever:
- From the paper: GPT-4.1 scored 14.6% with BM25 search vs 93.5% when handed the answer documents directly
- Same LLM, better retriever: 60% → 88% moving from BM25 to a reasoning-tuned ColBERT-style late-interaction model
- A well-tuned BM25 baseline (Pyserini experiments) still gets 60→80% — cheap lexical search shouldn’t be dismissed
- Better retrievers also mean fewer search calls to reach the answer (dense > lexical on both axes)
- Takeaway Hamel presses on: multi-vector/ColBERT-style retrieval is looking close to Pareto-optimal across the board; Thakur agrees it’s clearly best on BrowseComp-Plus and plausibly generalizes
Grading long-form answers
- TREC-RAG-style evaluation decomposes answers into nuggets (atomic sentence-level facts that should appear), support/attribution (does the cited document actually back the claim), and source credibility
- After years of LM-as-judge, TREC-RAG 2026 is trying an agent as the evaluator, validated by correlating against human labels from the 2024/2025 iterations (Kendall tau)
- Early finding: agents are biased toward “partial relevance” while human annotators are decisive (relevant / not relevant)
Training data: existing sets don’t teach search
- HotpotQA tops out at two hops and was built by giving annotators the two Wikipedia pages first; NQ is realistic but easy — no multi-round search needed
- Prior synthetic efforts: WebDancer (crawl site graphs, generate Q/A per page cluster) and E2HQA (iteratively add constraints to a question until it’s hard), Infoseeker (entity-graph constraint satisfaction) — useful but with prerequisites (crawls, entity graphs) that are cumbersome and expensive
- ORBIT (his project): hard training data with zero prerequisites and zero API spend
ORBIT’s recipe
- Seed creation: pick domains from the top-100 Wikipedia categories, collect the associated pages
- Generation: feed each seed page to DeepSeek’s free web chat (driven one-by-one through a Selenium instance — he had no budget) to produce a hard multi-hop question + answer
- Self-verification: the same model re-checks its own Q/A pairs; keep only confirmed ones
- External verification: scrape the cited sources and require open-weight models (a Qwen3-4B and a 12B-class model) to reproduce the answer from those documents alone
- The goal is zero false positives over coverage — many pairs get filtered, but what remains is multi-model-verified. The external stage removes the self-bias problem (a model that “remembers” its own generated answers) at the cost of dropping the hardest examples — a deliberate trade-off
- Result: 20,000 verified questions spanning beyond Wikipedia (ScienceDirect, Britannica, Stack Exchange verification links), needing more reasoning hops than HotpotQA (e.g. “the runtime of the 2017 animated film set inside a smartphone directed by a filmmaker known for sequels to children’s franchises” → The Emoji Movie, 86 minutes)
- Validation was honest but scrappy: he eyeballed 50–200 random pairs at each pipeline stage until answers were accurate
Training: RL with the simplest possible reward
- Standard recipe is RLVR (reinforcement learning with verifiable rewards): binary reward — 1 if the final answer is correct, 0 otherwise. A simple answer-level reward is a strong baseline; Search-R1 used GRPO with exact-match reward on NQ + HotpotQA
- ORBIT-4B trains Qwen3-4B on the ORBIT data; the agent trajectory looks like think → search → read → answer, repeated
- Same base model and same training pipeline across data sets — only the data changed — and ORBIT wins on Wikipedia-dump evals, most convincingly on Frames, the least saturated benchmark: it’s recent and genuinely multi-hop (verification often needs 5+ documents), unlike MuSQ’s one-or-two hops
The leaderboard trap: 95% accuracy, 210 search calls
- Top of the BrowseComp-Plus leaderboard: 95% EM / 93% recall — with 210 search calls in its trajectories. It burns enormous reasoning tokens to get there
- A model at 90% accuracy / 84% recall in 12 search calls is the better agent in his book
- EM accuracy alone doesn’t tell the story — you need trajectory-level inspection: how many turns, repeated queries, new information per document
HawkEye: seeing the trajectory your agent took
- Under-review visual analytics tool (built with his UWaterloo labmate Christina) for inspecting agentic-search trajectories
- Per-query signals: topic shift (are consecutive searches exploring new topics?), query provenance (does the query come from the original question, the retrieved context, the model’s own memory, or a previous query?), and document novelty (fresh information vs re-retrieving what’s already in context)
- Analysis of agentic-search logs shows heavy repetition: agents re-ask the same semantic question reworded (the “declarative” class dominated real query logs), looping when they can’t find something relevant — wasted tokens, invisible in an EM score
- Correct runs need far fewer search rounds than unresolved runs — the view that tells you where to set your own max-search-rounds threshold
- Supports side-by-side agent comparison (and retriever A/B with the agent fixed), and you can eyeball trajectories yourself — no clustering model required
Building benchmarks on private data
- Enterprises universally want a benchmark but sit on private data they can’t expose; historically retrieval benchmarks assumed human queries, but today LLMs generate the queries — even scraped Google logs can’t reliably tell human from LLM queries anymore
- His view: benchmarks need both — human queries (ambiguous, short) and LLM queries (verbose context that makes the retriever’s job easier); synthetic-query benchmarks like ObliqueBench (cluster documents sharing a research problem, generate a synthetic question that links them, require the agent to retrieve all of them) are the direction
- Search isn’t just web QA: coding agents retrieve files and repo context constantly — same evaluation and trajectory lessons apply
“If you substitute a good retrieval model with a bad one, you may actually lose out on the EM accuracy for your search agent.”