Brussle asks whether a model that says “80% sure” is right that often—not merely whether it chooses a valid answer. Its updated evaluation adds pinned jev-1.14.0 to a comparison with gateway-served Jev, SubQ’s preview model, and GPT-6 Luna.

I read the evaluation, methods, error analysis, and appendix. The measurements and statistical findings are Brussle’s, not independently reproduced here. Brussle sells a product that corrects decision probabilities, so its correction results are also a product evaluation.

Calibration depends on the question

Brussle reports 1,500 human-labeled questions from 21 public datasets, split across yes/no, category-choice, and ordered-scale questions. Its expected calibration error (ECE) measures the average mismatch between confidence and correctness across confidence bands; lower is better.

  • Yes/no: Jev 1.14.0’s ECE is 0.023, passing Brussle’s finite-sample calibration criterion.
  • Category choice: its ECE is 0.098, and its highest-confidence band averages 0.985 probability but 90.0% correctness across 339 answers.
  • Ordered scales: its ECE is 0.275; none of the tested models passes this question type’s calibration criterion.
  • SubQ’s preview is the only model passing the overall category-choice check; that does not make it calibrated on every subset.

These are probabilities for the selected answer, not the API’s separate confidence statistic or a guarantee about a weighted numeric score.

The average hides the cases that need caution

On the 100 hard yes/no questions, Brussle reports Jev 1.14.0’s ECE rising to 0.222, above that slice’s 0.115 calibration threshold.

For ordered-scale answers assigned probabilities between 0.7 and 0.9, the same model averages 78.6% confidence but only 34.5% correctness across 119 answers in Brussle’s results. The lesson is not “never use probabilities”; it is to validate the question type and the slice where your automation threshold will operate.

A correction is not a better underlying decision

Brussle fits a probability correction on one half of the labels, tests on the other, then swaps halves. It reports Jev 1.14.0’s ordered-scale ECE falling from 0.275 to 0.038.

  • Ordered-scale and category-choice winners stay unchanged: a less confident wrong answer is still wrong.
  • The questions, labels, and source rows are private, preventing an independent reconstruction from the page.
  • Public source datasets leave possible training contamination; this is not a fresh production-outcome test.
  • Direct TypeSafe and gateway routes differ, so reported latency differences cannot be attributed solely to a model upgrade.

Brussle’s broad “use as they come” wording goes further than the evidence warrants for a new application. Validate probabilities on your own outcomes, keep difficult-case checks separate, and never treat a typed answer as a correct or authorized decision.