Hamel Husain answers who should own AI evals, the tests used to judge an AI application’s behavior. A three-minute explanation on his channel.
Keep evaluation close to building
- Everyone building the application should inspect its data and failures.
- A separate evaluation role can become a silo, cutting quality decisions off from product development.
- Assign responsibility while keeping evaluation inside the team’s regular improvement loop.
Let domain expertise lead
- The person who understands the work should drive error analysis and decide what good output looks like.
- Developers are domain experts for coding applications; other products may need a different expert.
- A product manager’s or engineer’s title alone does not establish that expertise.
Give one trusted expert the final call
- Husain recommends a “benevolent dictator”: a domain expert the team trusts to settle quality judgments.
- Many annotators and repeated disagreements can make labeling expensive.
- Aim for signals that guide improvement; a single decision-maker is a practical default when the domain permits it.
Make reviewing data easy
- Build annotation tools that display the example and its relevant context together.
- Tailor the interface to the actual data and domain; Husain suggests using coding agents to build it.
- His recommendation is an organizational starting point, not evidence that one reviewer suits every risk level.
“You’re not trying to get to perfection, you’re just trying to extract good signals from your data so that you can improve.”