Yinheng Li and Justin Wagle ask whether you need a special decision-model architecture at all. Their LLM-as-Jev method puts numbered choices in a prompt, then reads the probabilities of their bracketed answer IDs from an existing language model. It does not use TypeSafe’s proprietary Jev weights or API.

The method

  • Present all choices together as [1] … [K]; score each possible completion after Best answer: [ and normalize those scores into a distribution. A closing bracket keeps IDs such as 1] and 12] distinct.
  • Share the prompt computation across candidate continuations instead of generating a paragraph or running a separate prompt per choice. The paper also proposes optional fine-tuning with a choice-ranking loss and KL anchors that discourage damage to the model’s other text predictions.

What the authors found

  • Their unmodified Qwen3.5-4B answered 188 of 231 public JevBench items correctly (81.4%), against 186 for their rerun of SemIf on the same backbone; the small difference is not statistically significant. These are the authors’ runs, not reproduced here, and not results on the sealed test set.
  • With a larger set of standard classification and reasoning tasks, the numbered-answer method did better than SemIf’s letter-token readout. On the 77-choice Banking77 intent task the frozen model scored 69.0%; targeted tuning lifted it to about 75% for the 4B model.
  • Training only on intent data made the 4B model too eager to answer “yes,” avoid “none of the above,” and match surface words. Its public JevBench accuracy fell from 81.4% to 74.9% after an epoch even while training loss improved.
  • LoRA with anchors reached 84.0% on those public JevBench items in the authors’ run. Removing the anchors caused some models to keep generating text after the choice or fail to end chat responses, even when aggregate benchmark accuracy looked similar.

Takeaway: start with a frozen-model baseline, evaluate on your own labeled decisions, and tune only for a demonstrated gap. A well-formed probability distribution is not proof that the selected action is right. The public benchmark also influenced the authors’ data-mixture revisions, so its post-tuning scores should not be mistaken for a sealed holdout result. See the broader Jev benchmark for task-specific calibration failures.