Sangdo Han and Moonsoo Kim test whether a second grading model from another provider makes an AI comparison more trustworthy. Their result supports a sensitivity check, with a substantial cost in unanswered comparisons.
Five judges assess 4,000 human-voted answer comparisons involving nine systems and 33 system pairs. Each sees both answer orders. The judges produce substantially different rankings despite reading the same answers.
- Among 23 statistically clear human system comparisons, Haiku and Gemini Flash-Lite each match the human conclusion on 22. Their agreement preserves the same shared mismatch.
- GPT-6 Luna produces significant conclusions for 14 pairs, matching humans on seven. Requiring agreement with either other low-cost judge leaves only eight comparisons, seven matching the human conclusion.
- The average precision gain therefore comes from screening out a poorly aligned judge. Agreement between the two already-aligned judges adds nothing.
The reference is a single person’s preference for each answer pair, not verified correctness. Shared disagreement with a voter can reflect voter noise as well as shared judge bias.
Execution settings also matter. Luna runs with reasoning disabled, while larger Claude judges use low effort. The authors dropped a planned sixth judge after seeing partial confirmation results and changed the majority rule after the freeze. Both deviations are disclosed in the released analysis bundle.
These are author-reported results on older Arena answers. For an evaluation workflow, report coverage alongside precision and check representative human examples. Agreement across providers alone cannot certify a conclusion.