欧的Leo asks whether a local assistant needs a second model just to make bounded decisions. Rather than generate and parse a reply, the author reads the existing model’s scores for answer labels directly. The experiment tests local alternatives—not TypeSafe’s hosted Jev.

I read the full writeup; I did not reproduce its benchmarks or inspect an accompanying implementation. All results below are the author’s measurements.

Read the choice instead of generating it

  • Frozen Qwen and Gemma weights receive context, a question and enumerated answers, without extra decision training or a new classifier head.
  • One forward pass supplies the answer-label logits—the model’s raw preference scores—rather than a generated sentence.
  • This still runs the full model over the input. It saves subsequent generation, not all inference work.
  • Label preferences are not calibrated probabilities that a decision is correct.

A useful local baseline, not a Jev verdict

  • On the author’s 300 fixed inputs, Gemma direct scoring reached 75% accuracy, versus 77% for OpenDecider small and 61% for Qwen direct scoring.
  • The author reports median warm times of 259 ms for Gemma and 707 ms for OpenDecider on an M2 Max with 64 GiB memory; loading and context preparation were excluded.
  • Different model sizes, quantization and input formats prevent a clean test of whether specialist training helps.
  • Laya Typed Decisions scored about 77% on the public dataset and 51% on the author’s tasks, but changing tasks, language and input conditions cannot establish overfitting.

Count the whole workflow

  • A separate same-Gemma comparison favored direct scoring, but also changed runtime, caching and the parsing/repair pipeline.
  • The author reports about five seconds to prepare direct-scoring context on a cache miss; warm request time alone hides that cost.
  • Both general-purpose baselines struggled to recognize requests that should not be forced into the offered choices.
  • One-model versus two-model workflows were not compared end to end: routing, fallback, memory residency and maintenance remain open questions.

The practical lesson is to benchmark a direct-scoring path in the model you already keep resident before adding another service. Preserve an escape hatch: a valid label is neither proof of correct judgment nor proof the task belongs in the menu.