How Glean Evaluates Agent Retrieval

Manav Agrawal and Lauren Shrack of Glean’s search quality team walk Hamel Husain through their evaluation and launch process.

Build datasets around the product

Public benchmarks miss enterprise permissions, diverse private sources, and signals such as document freshness and authorship.
Mine internal search clicks and assistant citations as imperfect labels for documents and snippets; retain original chat questions for end-to-end tests.
Generate and filter question–answer pairs from important documents, using real queries to guide synthetic examples.
Create targeted slices for particular features, and keep training and evaluation data separate.

Track recall at the number of documents the assistant receives, often 8 or 16; also track ranking position and changes to the result order.
For context selection, measure whether the relevant snippet survives while keeping token use manageable.
Use pairwise completeness judges when a canonical answer is absent, and correctness judges when one exists; check citation support and missed relevant sources.
Track tool success, token use, latency and agent loops alongside answer quality. Model judges and historical clicks are proxies, not guaranteed ground truth.

Engineers include analysis of roughly five to ten changed examples, including wins and losses, in launch documents.
Start with cheaper search evals, then run end-to-end chat evals and, for appropriate changes, live A/B tests.
User feedback is sparse: votes, copies and follow-up behavior are useful signals, but need careful interpretation and adequate sample sizes.
A missing logging signal for Slack threads reduced citations; per-source monitoring caught what offline tests had missed.
Dedicated teams support the infrastructure and evaluation tools, while feature engineers own their evaluations and regression sets.

Synthetic tests can saturate as models improve; click-based sets inherit exposure bias from earlier rankings.
The team is working on more representative golden sets, reproducible write actions and multi-turn simulations.

“The best way to sort of learn what evals is trying to construct an eval set on your own.” — Manav Agrawal