Hamel Husain and Isaac Flath test Anthropic’s automated evaluation-building skill on real apartment-leasing conversations. A 48-minute live walkthrough, not a controlled benchmark.
What Claude gets right
- It identifies genuine transfer and scheduling failures in a dataset Hamel already knows well.
- It notices voice and text have different instructions and retracts some suspected pricing errors.
- Its advertised principles are sensible: realistic tasks, room to improve, stable grading and human review.
Where the workflow goes wrong
- It recommends a test before the humans have inspected the conversations or compared user impact.
- It asks reviewers to check model-generated labels in Markdown, with truncated tool results.
- It initially presents aggregate scores rather than the individual evidence needed to challenge them.
- A usable review page appears only after Hamel explicitly asks for one; annotation persistence remains fragile.
The grader tests the wrong things
- Literal confirmation wording is brittle: callers can request a human without using a particular phrase.
- Repeated questions, failed transfers and awkward narration are different problems, not one pass/fail bundle.
- A bot promising a transfer without making the tool call is more consequential than a verbose confirmation.
- Agreement on “fail” is not enough if the grader gives the wrong reason.
The better starting point
- Build a readable viewer that preserves messages, instructions, tool calls and tool results.
- Read real conversations and write your own observations before seeing suggested verdicts.
- Prioritize failures by frequency and user harm; fix obvious bugs without automatically creating a new eval.
- Build focused tests, validate their case-level reasoning, then optimize against scores you trust.
Both speakers reject this workflow in the demonstrated run, while acknowledging better failure discovery than the earlier tool they tested. They do not demonstrate the separate hill-climbing stage, and they are testing recorded traces without application code.
“You don’t know what good looks like and it’s really helpful to look at some data before you jump into evals.” — Hamel Husain, 42:17