iwhalen.com tested whether Jev can act as a search-relevance judge: given a query and passage, choose a human-style grade from irrelevant to exactly answering the question. The interesting result is not a simple win. Agreement with individual human labels was poor, yet the induced rankings of search systems were much closer.

Two different tests

  • On the author’s TREC DL 2023 judged-passage dataset (22,327 query–passage judgments), Jev’s reported Cohen’s kappa against human grades was 0.1907; GPT-6 Luna’s was 0.2391. Neither is strong individual-label agreement. These are the author’s results; I did not rerun the paid model calls.
  • The author also regraded retrieval runs and compared their search-system rankings against rankings from human labels. On that relative ranking task the article describes Jev and Luna as essentially tied; the chart does not supply a numerical Kendall correlation in the text. This is not proof that Jev assigns correct relevance grades.
  • In the author’s parallelized run, estimated total model costs were $0.6783 for Jev and $1.3283 for Luna, with mean per-request times of 0.145 s and 3.442 s respectively. Per-request latency is not total wall time; Luna’s cost is an estimate from uncached token usage.

Why the method matters

  • The repository publishes the dataset-building and evaluation code: same four-grade rubric for both models, durable per-pair checkpoints, confusion matrices, and retrieval metrics computed from the substituted judgments.
  • Jev returns a typed choice and probabilities; the code takes the highest-probability grade. A well-formed grade is not the same as a grade a human would endorse.
  • Caveats: one dataset, one prompt formulation, no local/open-model comparison, and no demonstrated production search improvement. The author explicitly says the prompts may need tuning and cautions that unjudged passages are treated as irrelevant.

This is a useful case for optimizing the downstream decision you actually need—ranking systems—without mistaking it for reliable grading of each document. Related: Jev Across 37 Datasets tests a different classification benchmark and its thresholds.