ankur287’s jev-open-calibration compares small, locally trained classifiers with TypeSafe Jev on banking customer-support messages. It publishes implementation code and raw API responses, making the confidence claims checkable offline.

Calibration measures whether stated probabilities match observed correctness. A model can choose the right category while giving that answer a misleading probability.

Compare the training budgets

The repository reports these results on BANKING77, a dataset of messages labeled by customer intent:

  • On its smaller task, both the temperature-scaled local classifier and Jev answer 97.5% correctly. Local checks reproduced Jev’s 273 correct answers from 280 saved responses.
  • Across the full task, the author reports 88.8% accuracy for the supervised local classifier. Jev reaches 80.1% with terse category descriptions and 85.9% with examples.
  • Offline checks reproduced both Jev accuracy figures from the saved responses. The local classifier’s training results were inspected, without retraining.

The local model learns from labeled training messages. The stronger Jev prompt includes three examples per category. These are different ways to supply task knowledge, so this comparison supports a workload-specific baseline.

Change the probabilities without changing the labels

Temperature scaling adjusts how sharply a classifier concentrates probability on its preferred answer. The implementation fits one scaling value using predictions from training folds held out of each fit.

The author reports expected calibration error, the average gap between confidence and accuracy, falling from 0.248 to 0.031 on the full task. The predicted labels stay unchanged.

  • In the author’s cascade experiment, a 0.90 cutoff accepts 21.1% of messages before scaling and 78.9% afterward.
  • The assumed reviewer accuracy and escalation costs are inputs to that simulation. They are not measured production savings.
  • The local backend handles a fixed Choice classifier. Its question descriptions do not teach it new categories at request time.

Give the schema a way to reject the request

The repository also tests messages whose true category is missing from the offered choices. In Jev’s saved sample, 26 of 100 cross the 0.90 auto-action cutoff despite having no valid answer available. That count reproduced offline.

The author reports the same failure pattern for the local classifier, using a different sample. Better calibration on familiar categories leaves this missing-answer problem unresolved. Typed output guarantees an answer’s shape; correctness still depends on the question and available choices.

Jev’s yes/no framing also changes the operating point. The saved results disagree with Choice about whether to act on 111 of 280 messages at the same cutoff. Offline checks reproduced that count.

Keep the evidence boundary visible

The matched-bin analysis uses 15 equal-width bins for calibration comparisons. Local recomputation matched its Jev figures. The author’s paired uncertainty interval crosses zero on the smaller task, leaving the claimed calibration ranking uncertain.

One saved response selects a category with probability 0.42 while another has 0.43. That small inconsistency barely changes aggregate calibration error, but warrants checking the returned label against its scores.

Verification: inspected the README, local backend, calibration and experiment code; recomputed Jev metrics from author-provided artifacts. No live Jev calls or model retraining were performed. BANKING77 is public, so possible training overlap remains unknown.