Hamel Husain on his own channel — a 2-minute clip on why traditional scoring metrics (ROUGE/BLEU) and rating scales make evals harder to act on, and why he defaults to binary pass/fail.
ROUGE/BLEU: string similarity from a different era
- These scores come from traditional ML/NLP — at their core they measure string similarity
- A coarse metric that made sense when LLMs could barely produce coherent language; “we’re way past the point of producing coherent English” now
- Part of the “eval industrial complex”: we have metrics, trust us because we’re experts — it feels fancy but isn’t what you need
Rating scales hide uncertainty
- Unless you have a very sophisticated setup and put real resources into aligning scores with humans — which ~99% of teams don’t — a 1-to-10 or 1-to-5 scale is a bad idea
- Nobody knows what 4.2 versus 3.7 means
- Humans just hide their uncertainty in the middle values, so you don’t get good decision making out of it
Binary pass/fail is what you can ship on
- At the end of the day you have to ship your product: is this good or not?
- Binary evals are easier to align with humans — which you always have to do with an eval — and easier to action
- A score of 3.2 → “I don’t know what’s wrong.” A fail → “okay, it failed, now you can action on that”
“You see a score of 3.2, you’re like, I don’t know what’s wrong… but fail, it’s like, okay, it failed, now you can action on that.” — Hamel Husain