Hamel Husain explains why annotation work should usually begin with open-ended observation rather than a premature rubric or “correct” reference answer.
Start without the answer key
- A reference answer is an example of a correct response.
- A rubric is a set of criteria used to judge a response.
- Neither is required before inspecting data; defining them too early can bias annotators toward the team’s initial assumptions.
- Husain calls the discovery of better criteria while examining real examples criteria drift.
Let traces reveal the problem
- Imagine a support agent that follows the written refund policy and escalates correctly.
- Reviewing traces may reveal that the escalation itself is clumsy and frustrating, even though it satisfies the original specification.
- An early rubric focused only on policy compliance would miss that product failure.
The sequence that works
- Review representative traces without forcing observations into preset categories.
- Write open-ended notes about anything confusing, incorrect, or harmful.
- Group similar notes and count them to expose recurring patterns.
- Turn the observed patterns into a stronger rubric and better reference examples.
“It’s often better to wait until after you look at some data before you specify a rubric or have reference examples.” — Hamel Husain