Hamel Husain on his own channel — a 1.5-minute clip on LLM-as-a-judge evals: how to measure the judge’s noise and why a judge you can trust 100% of the time doesn’t exist.

Measure the judge against ground truth

  • An LLM judge is a “soft check” — you can scientifically measure how good it is by comparing it to labels
  • Assemble a dataset of human labels as ground truth (is this thing good or bad?) and measure how much noise the judge has
  • Even ~100 labels is enough to measure the noise
  • There are different kinds of noise — you can tune the judge to reduce it, but there will always be some

A judge is a black-box classifier

  • There’s no such thing as a judge you can blindly trust with no noise at all
  • It’s a classifier, like a stop-sign detector: get it very reliable and there’s still an edge case somewhere producing false positives and false negatives
  • You have to be okay with that

Tolerable noise is a business decision

  • Tune the judge to reduce noise, then make the call: is this level of noise tolerable for your use case?
  • Bake the eval’s known characteristics into your decision making — that’s the best you can do with an LLM judge

“Even if you can get that to be very reliable, there’ll always be an edge case somewhere where it’s going to give you a false positive, false negative.” — Hamel Husain