Synthpop’s controlled audit separates two ideas that are easy to conflate: confidence that tracks accuracy on average, and confidence that falls when the model has no basis for an answer. Across 575,442 calls to Jev 1.13.0, the first often worked; the second did not.

Confidence worked on familiar ground

  • On converted TriviaQA questions, keeping answers with at least 90% confidence retained 8,132 of 9,960 questions and only 20 were wrong.
  • With real category names or enough labeled examples, Jev’s confidence generally rose alongside accuracy on Banking77 and CLINC150.
  • The authors report low calibration error on several familiar benchmarks, although MMLU-CF—a contamination-resistant set—showed 91% confidence at 78% accuracy.
  • These are the study authors’ results. I reviewed the article, paper metadata, and open experiment repository but did not reproduce the 575,442 paid API calls.

Remove the knowledge and confidence stays

  • The researchers replaced 77 or 150 intent names with opaque codes, then varied only the information supplied about those codes.
  • With no hints, Jev was right about 1.0% on Banking77 and 0.8% on CLINC150—roughly chance—but averaged 32.0% and 36.3% confidence in its top choices.
  • An open comparator, GLiNER2.5-Decide, stayed near an even split without answer-relevant information, though it was less accurate than Jev when real category names were present.
  • On a fair-die question, Jev picked “one” in all 720 option orders and assigned it about 80% probability on average, despite every face being equally likely.

The natural knowledge boundary was worse

  • On dated news questions, the authors found an observed accuracy change around November 2024; they explicitly do not claim this is Jev’s training cutoff.
  • Before that boundary, Jev was reportedly 70% accurate at 75% average confidence.
  • Beyond it, accuracy fell to 51%, while average confidence rose to 82%.
  • Jev answered “no” to 92% of later questions, behaving as if an event it had not encountered had not happened.
  • Recalibration on earlier months did not repair the shift: later questions remained about 20 percentage points overconfident.

Why this matters for automation

  • A typed response guarantees that the answer fits the offered schema. It does not establish that the chosen option is correct.
  • A threshold gate needs case-level uncertainty: missing knowledge should lower the score before software acts automatically.
  • Aggregate calibration can hide exactly the failures a review gate is meant to catch.
  • The safe pattern is to validate on held-out, application-specific data and test explicit evidence-sufficiency questions—not to treat a plausible probability as self-knowledge.

Important limits

  • The audit covers one reported model version, jev-1.13.0; later versions may differ.
  • Most free-text benchmarks were converted into multiple-choice tasks, which are easier than open response.
  • The observed news boundary mixes time with possible changes in topic, style, and difficulty.
  • Date-removal labels were provisionally produced by an AI assistant and had not yet received independent human review.
  • The paper is an arXiv preprint, not formal peer review; the article says a disclosure about provider relationships was still pending author confirmation.

“Jev’s confidence is useful on familiar ground and unreliable beyond it.”

The practical conclusion is narrower and more important than “Jev is badly calibrated.” Jev can be calibrated on a known distribution while failing to signal when the distribution—or its own knowledge—has changed. Those are different properties, and production confidence gates need both.