Cameron Wolfe, a staff research scientist at Netflix, outlines a three-stage evaluation loop for subjective AI tasks in this 18-minute AI Engineer talk.
Define quality with people first
- Work with domain experts to choose a small set of quality criteria and clear rating scales.
- Give reviewers examples, compare their judgments on the same outputs, and revise ambiguous guidelines.
- Use human-to-human agreement as a realistic reference when assessing an automated judge.
Build an automated proxy you can check
- Use a language-model judge to apply shared guidelines, or a rubric of task-specific checks.
- Test that judge against trusted human-labeled examples before relying on its scores.
- Wolfe describes a Netflix project linking judge scores to viewing abandonment; the talk supplies no effect size or study design.
Measure uncertainty alongside scores
- Both the agent and its judge can vary between runs, making small score changes hard to interpret.
- Use power analysis to plan sample sizes around the difference you need to detect and the noise you observe.
- Report uncertainty intervals and sample repeated agent or judge outputs when estimating variability.
- Choose realistic, diverse tasks that distinguish model capabilities, and keep improving the evaluation data.
Close the loop in production
- Screen many ideas offline, then send promising candidates through human evaluation.
- Use A/B tests for a staged rollout, tracking explicit feedback and relevant behavioral signals.
- Feed new failure cases and changing use patterns back into the guidelines and offline tests.
Caveat: this is an evaluation roadmap, not a reported Netflix model comparison. It gives no universal sample size or judge-agreement threshold.
“Human evaluation serves as our North Star for quality.”