Shreya Shankar (evals researcher, taught the evals course now at 4,500+ students) and Hamel Husain on Peter Yang’s channel — 54 minutes of live demo: using Claude Code to turn your taste into evals, plus Hamel’s benchmark of the “auto-eval” vendor tools.

Evals still start with data

  • The fundamentals haven’t changed: look at data first, do error analysis, externalize your taste and judgment before writing any eval
  • What changed: agents are now good enough to help you look at the data — running in the background while you review, giving you leverage in that first stage
  • They’ve become bigger fans of LLM judges: an LLM judging a trace against one very specific, well-defined criterion (too long? too short? follows structure?) is now quite accurate

Top-down vs bottom-up evals

  • Top-down: from the task description alone — what makes good output? Word length, action verbs, actionable takeaways. Claude is very good at generating these
  • Bottom-up: discovered by reviewing many sample outputs — your gut vibes and feedback externalized into criteria. Claude is very bad at coming up with these. That’s all you. And it’s why they accumulate over time
  • Peter’s podcast-takeaway skill is the live example: he has the top-down half (character-length checks, “understandable without watching the episode”), but the bottom-up half is where the question mark lives — is it exhaustive of all the feedback he’s given across every episode?
  • His loop: run the skill → go back and forth → “reflect on our entire conversation and update the skill and evals so we don’t have to do this again” — with the honest worry that it overfits (one interview’s MECE complaint may not matter for the next)

Three practical tips for eval-heavy skills

  1. Separate your evals into top-down and bottom-up inside the skill itself
  2. Fan out to sub-agents: with lots of criteria, give one sub-agent one criterion (or group) — give a model the whole list and it gets lazy and ignores things; focus it on one piece and it really focuses
  3. Have the AI write a spreadsheet / pivot table of criteria × pass-fail indicators, so you can see the hierarchy yourself and make judgment calls on what matters for this particular case

The error discovery skill (the live demo)

  • Open-source, free (link in the episode description) — invoked from Claude Code; it built the whole review interface from scratch in ~15 minutes, live on camera
  • Five steps:
    1. Read the dataset and figure out its semantic type (article? code? traces?)
    2. Design a visual encoding — color, spacing, opacity (Gestalt principles) to show what varies in the data
    3. Build an interface — an HTML review app (Python backend); “so much better than me looking at my data in Google spreadsheets”
    4. Pick which samples you should look at — clustering, diverse initial sample
    5. Interactive loop — the agent watches your in-situ feedback via the monitor tool and proposes new samples or rubric criteria in real time
  • Design philosophy: the human reads and gives open-ended feedback; the agent’s job is not to invent feedback but to group and distill it into actionable rubric criteria
  • The writing demo: he reviewed AI-generated articles and gave taste feedback — “I don’t like negative contrast (‘it’s not X, it’s Y’)”, “I hate the list of threes”, staccato fragments — the agent annotated 361 suggestions across the dataset, and the most frequent failure mode was staccato fragments (he’d have guessed negative contrast)
  • Live reflection beats reflect-at-the-end: interleaving human think time with AI think time, and the ~10-notes threshold works as a “carrot” that makes you actually read samples
  • Once the rubric exists: turn it into a skill, one LLM judge per criterion, a dashboard, or live monitoring — the hardest part of evals is error analysis, and this automates the discovery half
  • Bonus: how you eval something should inform how you design its interface — the same failure-mode annotations that power evals would make a great IDE that flags staccato as you write, instead of silently rewriting

Do automated evals actually work? (Hamel’s benchmark)

  • Vendor “auto-eval” tools (BrainTrust, Arize, LangSmith) promise: upload traces, chat with an AI, get your evals done
  • Benchmark vs a human-annotated dataset: the tools recover a lot of the errors a human would — but all of them miss the same thing: errors that require product judgment and taste (e.g. a rental bot that doesn’t handle sales objections, or markdown leaking into text messages)
  • Coding agents (Claude, Codex) performed about the same — the harness is thin; it’s someone else’s prompt
  • The real benefit of the vendor tools is integration into your stack (traces in LangSmith → use LangSmith); precision is 80–90% best case, so 10–20% of “errors” found are red herrings — check recall AND precision, and sanity-check what the tool found
  • Bottom line: automated tools get you a good baseline; manual review of the data is the edge — “actually reading stuff” is the edge, in evals and in code

“There is no world in the future — even if you have AGI — if you’re building a product, you have to look at your data. You have to be able to inject your taste into the development of your product.”

The course they teach (five modules, revamped for agentic AI): error analysis philosophy, productionizing and scaling evals across a team (aligned criteria, CI/CD, drift monitoring), safety and adversarial evals (tenant data leakage), and cost/accuracy improvement — they claim 100x cost cuts on real consulting cases by evaling your way down to cheaper models.