Ivan Yee Lee and Taylor Berg-Kirkpatrick test a deceptively simple alternative to a specialized decision model: stop an ordinary open-weight language model just before it answers, then read the probabilities it assigns to the allowed answer labels. It makes a closed choice in one forward pass, without writing prose. A typed choice can still be wrong.

The comparison

  • Their benchmark contains 19,505 decisions across text classification, agent monitoring, and triage. The overall score gives those three families equal weight, even though triage has far fewer examples.
  • By their three-point non-inferiority rule, seven of 23 ranked open models matched Jev on overall macro-F1 (a score that gives rare classes weight). Jev scored 0.787; Gemma 4 31B-it scored 0.794, but the measured difference includes zero. These are the authors’ results; I did not rerun the models.
  • The aggregate hides a split: six of the seven outperformed Jev on agent monitoring, while Jev outperformed six on triage. Changing guardrail labels or the beer-record matching rule moves the triage comparison.

The practical tradeoff

  • Jev’s returned probabilities were better calibrated as delivered than six of seven alternatives. Fitting a temperature on separate labeled examples substantially improved the open models; Jev’s probabilities are rounded to hundredths, making its own calibration comparison sensitive to how exact zeros are handled.
  • Through hosted APIs, the accepted open-model timings were slower than Jev’s stored calls. One locally served Gemma configuration was faster than Jev’s hosted calls without network time—not an apples-to-apples latency win.
  • A cheap hosted listing is not necessarily the same precision, quality, or log-probability interface that the authors tested. At median OpenRouter listing prices, none of the matching models was cheaper than Jev in their comparison; a busy rented card changes the economics.

The interface—bounded options plus probabilities—is reproducible with an ordinary model. Whether it is useful depends on labeled examples from your own workflow, calibration, hosting, and the cost of a wrong answer. The source is an independent study, not proof of Jev’s architecture. Article and methods reviewed; models not run. Related: Cisco’s safety-threshold comparison and Raschka’s classification study.