Hamel Husain offers a beginner’s guide to evaluating an AI product, not choosing a model from a general benchmark. The order-cancellation examples are illustrative, not a test of a named commerce system.
Observe the product before writing graders
- Record complete traces: user input, prompts, context, tool calls and results, assistant response and any subagents. A failure can hide between a failed cancellation call and an assistant saying the order was canceled.
- Read and label examples, group failures by mechanism and count them. Keep some random exploration even after clustering and metric-driven sampling expose recurring patterns.
- Synthetic inputs can help fill underrepresented scenarios; run them through the actual application and review the resulting trace. Do not substitute model-invented complete transcripts for production behavior.
Calibrate checks against human judgments
- Use code assertions for checkable facts and narrow binary model judges for specific failure modes. Compare both missed failures and false alarms with human labels.
- Split examples into development and untouched final test sets; tuning a judge on the development set can overfit it.
- Decide whether a failure is observable, important and likely to require iteration before paying to automate its eval. A vague 1–5 “quality” score is rarely an actionable product rule.
Change the system, not just the score
- Use the checks to compare prompts, workflows or cheaper models against a minimum quality bar. Human-review agent-proposed changes and test them on unseen cases to avoid metric gaming.
- A product’s task-specific cost/quality frontier can differ from public foundation-model rankings. Measure whether your user task succeeds.
“The most important thing to know about evals is you should be using evals to make your product better.” — Hamel Husain