Shreya Shankar (Stanford CS professor, co-creator of the AI evals course with Hamel) kicks off the 12-part AI product engineering series. 27 minutes on Hamel Husain’s channel.

Why this matters

  • Output quality is the biggest barrier to productionizing agents (LangSmith annual report) — and figuring out how to evaluate models is genuinely hard
  • Vendors (LangChain, Braintrust, Arize) are selling end-to-end automated eval tools: point an LLM at your traces, it finds and fixes your bugs
  • The catch is epistemic: what “good” means lives in your head, not in the traces — if a tool could fully fix your product, it could fix everyone’s, and there’d be nothing left to differentiate yours
  • AI’s real job: help you express and apply your judgment faster, not replace it

The eval lifecycle (analyze → measure → improve)

  1. Error analysis — the hardest step: take traces and find failure modes. No perfect definition of “mistake” (you can’t define slop, but you know it when you see it)
  2. Measure — how prevalent is each failure mode? Pareto applies: ~80% of issues come from ~20% of failure modes — prioritize those
  3. Improve — fix the product: prompt instructions, model switch, fine-tuning. Iterate forever

AI is weak at the front (taste-specific error analysis) and strong at the back (measurement, prompt optimization, hill-climbing).

Mistake 1: ask a coding agent to “just evaluate my app”

  • Agents find some issues but miss everything taste-specific — anything not externalized in traces/prompts is invisible to them
  • They don’t find the highest-priority issues, and they fabricate priorities (“extremely repetitive voice, the biggest issue” — vague, probably wrong)
  • No reuse: every run re-reads the data and rebuilds failure modes from scratch — persist intermediates instead

The fix: the open-source error-analysis skill (built into his workflow):

  1. Agent reads the data, finds shared structure/fields
  2. Designs a visual encoding (color-by-role etc.) for human review
  3. Builds a review app with three views: trace viewer, map view (cluster of traces), progress view (tree map of failure modes)
  4. Clusters + samples representative traces so the human doesn’t review everything
  5. Human annotates in situ (highlight text → feedback); the agent monitors, builds a failure-mode taxonomy live, and goes for breadth (all failure modes) + depth (multiple examples each)

Demo reality: took ~5 min for the agent to build the app (33 essays, 21 samples); the agent doesn’t always follow instructions (“why aren’t you using the monitor tool?”); UI generation is still rough — it’s a tool for seeing your data, not a polished product.

Mistake 2: one pass over your data

  • Revisiting already-analyzed traces surfaces new failure modes — more traces in your head = better analysis (his “matters” trigger phrase only appeared deep in the process)
  • Outer loop (iterate over data) × inner loop (iterate on hypotheses within a data point)
  • Have the agent apply every new annotation to previously-labeled traces, then accept/reject its suggestions — but the agent shouldn’t invent new failure-mode types; that stays with you (validating agent taste is a bad experience)

Mistake 3: one uniform accuracy bar

  • Internal tools (Slack summarizers) don’t need perfect accuracy; customer-facing apps do
  • Reason about worst-case scenarios up front — give Codex/Claude Code your traces + app description and ask “what’s the worst that could happen to a user?” (sabotaged citations, leaked private info in a journalist’s output) — work backwards into guardrails

Closing notes

  • Upcoming research (with Amel and Antariksha Dasgupta): benchmarking automated eval tools on real data — surprisingly, general-purpose coding agents (Claude Code, Codex) find failure modes more exhaustively than dedicated discovery platforms
  • Hamel’s add: at the end of the day an eval tool is someone’s prompt + a bit of harness — inject your own domain expertise and customize the agent to your data
  • New evals course this fall (with Hamel); Shreya is hiring undergrads/masters/PhDs for her research lab

“Your judgment is the only differentiating factor of your product.”

Watch on YouTube — full summary in the vault note.