Hamel Husain answers a recurring evals-course question in this 3:16 explanation: can the model powering your application also judge its outputs?
Same model is usually fine
- The application and the judge do different jobs; sharing a model or provider is not automatically disqualifying.
- Husain recommends narrowly scoped pass/fail checks, rather than broad judgments about overall answer quality.
- Models can exhibit bias, but the practical test is agreement with human-labeled examples—not provider identity alone.
Validate before switching providers
- Check precision: when the judge flags a failure, how often is it right?
- Check recall: how many of the human-identified failures does it catch?
- Test on held-out examples—cases not used to tune the judge—to see whether the check generalizes.
- If agreement remains poor, try a different model; do not add another provider solely to satisfy a blanket rule.
Start capable, optimize later
- Begin with the most capable available judge to establish reliable criteria and agreement with human reviewers.
- Optimize model cost afterward, checking that the cheaper judge still meets the same standard.
- A second provider can add onboarding and operational complexity; weigh that against measured benefits.
“What ultimately matters is the alignment with human labels.”
Caveat: this is practical guidance, not proof that same-model judging is unbiased; the validation is only as informative as its labels and test coverage.