DoubtBench asks a better question than “did the top label match?” It converts NVIDIA’s HelpSteer2 ratings into 7,455 typed decisions, then checks whether a model becomes uncertain on examples where the human annotators themselves split.

I inspected the benchmark builder, scoring code, methodology, raw run metadata, and published result files. Its local test suite passed: 25 tests passed and one skipped. I did not rerun the paid 7,455-question model comparisons or perform a security audit, so all model results below are the project authors’ recorded runs.

What the benchmark measures

  • Five 0–4 Score questions judge helpfulness, correctness, coherence, complexity, and verbosity.
  • A Noul question asks whether a response is acceptable.
  • Choice questions compare two responses, with every pair repeated in swapped order to expose position bias.
  • A seven-level Score question records preference strength.
  • Individual annotator votes become soft labels rather than being discarded behind one rounded “correct” answer.
  • Metrics include accuracy, calibration error, Jensen–Shannon divergence from human votes, disagreement correlation, selective accuracy, and position consistency.

The source is pinned to a HelpSteer2 revision, generated artifacts carry hashes, harmful prompts are filtered by recorded rules, and public JevBench overlap is checked. Raw model answers are committed, so the scoring can be rerun without calling the models again.

The recorded Jev result

On the authors’ validation run, Jev 1.13 answered all 7,455 items with:

  • 48.2% top-answer accuracy;
  • 67.8 composite DoubtBench score;
  • 98 ms median and 183 ms p95 latency;
  • $0.0134 recorded cost per 1,000 questions;
  • 0.14 correlation between model uncertainty and human disagreement;
  • 62.0% accuracy on unanimous items, versus 40.4% on split items.

Among the four tested models, Jev had the highest top-answer accuracy. Claude Haiku 4.5 had the highest composite score—69.2 versus Jev’s 67.8—because its verbalized probabilities were less overconfident. Opus 5.5 and Sonnet 5.5 were run at low effort, and some questions were counted wrong after refusals, so this is a benchmark configuration, not a timeless model ranking.

The important negative result

No tested model’s uncertainty aligned strongly with human disagreement. The recorded correlations ranged only from 0.06 to 0.15, with Jev at 0.14.

That matters operationally. A confidence threshold is useful only if low confidence catches the cases that deserve review. Jev’s higher thresholds did select more accurate subsets—the recorded 0.9 threshold covered 10.6% of items at 78.6% accuracy—but that is not the same as detecting social or semantic ambiguity. The model can be confidently wrong where humans disagree and uncertain where they agree.

“No model hesitates where humans disagree.”

Read the headline carefully

  • The published costs and latency come from the repository’s recorded runs; I did not reproduce them.
  • Claude supplies verbalized probabilities, while Jev returns native distributions. That is useful but not a perfectly symmetric probability interface.
  • About 69% of the composite is indirectly driven by the five attribute tasks because the score is micro-averaged over all items.
  • HelpSteer2’s official rounded label can be a value no annotator chose, so even the human-vote reference reaches only 82.6% top-label accuracy.
  • The benchmark tests judgment over AI responses, not every routing, moderation, ranking, or business-decision domain.
  • A typed output guarantees a valid answer shape. It does not make the decision correct or the probability operationally calibrated.

Takeaway

DoubtBench is useful because it makes uncertainty falsifiable. Jev looks attractive when the job is a large volume of cheap, fast, bounded judgments: it was the most accurate model in this recorded comparison and dramatically faster and cheaper than the tested Claude configurations.

But the benchmark also blocks the easy marketing conclusion. Jev’s confidence is not a reliable proxy for human agreement. Before using confidence to automate decisions and escalate only “uncertain” cases, teams still need labeled data from their own domain, threshold-specific error rates, and a review of which mistakes the threshold fails to catch.