Lucas Rocha, an engineer at Brazilian edtech Nova Escola and alum of Hamel Husain’s AI Evals course, tells his evals rollout story backwards on Hamel’s channel — 34 minutes on going from a messy spreadsheet to four calibrated judges running in CI and on daily production samples.

The story backwards: evals today

  • Nova Escola (AI-generated lesson plans for Brazilian teachers) now runs evals with no tickets and no engineer in the loop: the pedagogy team drives Claude CoWork pointed at the evals project — memory files, system context, Hamel’s skills
  • Demo: “keep working on evals based on your context — what’s next?” → the agent reads memory, flags judge-calibration failure modes (human labels, TPR/TNR), proposes a new failure mode (activities lack a specific goal), and drafts judge prompts from false-positive/negative hypotheses
  • The right-judge-prompt skill catches a real bug: training data leaked into the dev split used to evaluate the judge → clean, rerun, and the valid-data-evaluator skill shows version 4 is worse — a report lands for the education team

Workflow and tooling

  • Lucas prefers Claude Code CLI for flexibility; his team uses CoWork; observability is Langfuse (wrapped its API in a skill before the official MCP shipped ~1–2 months ago)
  • Custom annotator review interface (thumbs up/down + notes flowing back into Langfuse) — now bootstrapped from the course’s slash-create skill; even spreadsheet annotation got replaced by quick interfaces exporting CSV
  • A company-wide skills marketplace (marketing: voice/WhatsApp, product, governance) plus Hamel’s evals skills: judge prompt writing, validating the evaluator, review interface — they skip error analysis (do it themselves) and synthetic data (plenty of product data)

The messy first attempt

  • One giant messy spreadsheet: lesson plans, rubric criteria (later failure modes), axial coding of the most common problems per criterion with frequencies
  • Some findings were trivially fixable without evals — e.g. the AI emitting two+ learning goals per lesson plan — fixed once understood; others still haunt them
  • Totally manual: engineer extracted a CSV from Postgres; work happened in Excel with Claude side by side

The two-annotator mistake

  • The course says one expert — the “benevolent dictator”; Nova Escola used two, who disagreed with each other more often than a coin flip on several criteria
  • Signal: the rubric was unclear — two people reading the same criterion were assessing different things
  • Fix: rewrite the rubric, relabel the dataset — a month (sometimes two) of work per round

The three barriers

  • Data: raw multi-turn conversation traces — tool calls, RAG retrievals, injected context — hard to extract and navigate
  • Methodology: error analysis, open/axial coding, judge building, labeling, calibration (splits, balanced data), analysis — many steps, each with traps
  • Judgment (the one that matters): “is this lesson plan good?” — engaging, motivating, mobilizing the right knowledge — can you trust a machine to regulate that?

Lowering the walls: three layers

  • Observability first: Langfuse traces — which tool was called, timing, retrieved text, final answer — a goldmine; also the store for best datasets and prompts
  • Natural-language access: Claude plugged into Langfuse (MCP or skill) so the pedagogy team reads production data without an engineer
  • Packaged skills: the methodology embedded — split ratios, calibration steps — so the team bootstraps with guidelines
  • Result: the methodology barrier lowers but doesn’t vanish — what remains is the pedagogical judgment itself, and that’s the job

Calibrated judges in production

  • 4 judges calibrated to production quality (good-enough TPR/TNR; a few failure modes still lack a trustworthy judge)
  • Role 1 — CI gate: every PR runs the judges; did the new agent version regress a criterion?
  • Role 2 — daily production sampling: started at 2% (too erratic), raised to ~80% of daily lesson plans in batch mode at night — cheap, stable drift signals
  • Model choice: GPT-5.4 judges, a cheaper mini model generates; always run the judge on the model it was calibrated with so TPR/TNR still hold
  • The curriculum-alignment judge is the troublemaker — first hill-climb candidate; evals also broke on a new subject (computational thinking) and caught regressions during the agent refactor (Anthropic managed agents vs the artisanal LangGraph one)

Takeaways: keep the domain expert at the center

  • Instrument first — no observability, no evals
  • Make the data reachable — MCP/skills so non-engineers can query the observability tool
  • Skills are a starting point, not gospel — “there’s somebody else’s prompt” behind every skill; Lucas redelivered the 8-class course to his team so they knew what to expect
  • The point is quality and trust: remove the barriers so the people who know the product stay in the loop

Q&A

  • Product impact: the chatbot used to cite curriculum codes that didn’t exist, misleading teachers — evals directed scarce resources (390 people, 6 in tech) at the real problems; a year later the product is “totally different,” and the failure modes have moved on
  • Sampling and cost: batch mode at night; ~80% daily sample for stable signals; judge on the calibrated model, generate on a cheaper one
  • Two annotators vs one: education is nuanced, but annotator friction is “tremendous” — worth finding who actually knows if it’s good; sometimes the rubric writer should be the labeler
  • Eval discipline: if a problem is trivial, just fix it — evals are expensive; a persistent, unfixable error is the best eval candidate

“If you find an error that’s very persistent and is not going away, you don’t really know how to fix it — that is a great sign. You need an eval.” — Hamel Husain