Nicholas P. Waltz’s Jeeves is an open Qwen3.5-9B decision model that can reason before returning a bounded choice, score or yes/no answer. It publishes weights, training and inference code, and evaluation scripts. The final response stays typed even when the model generates a private reasoning chain first.
Spend reasoning on uncertain cases
- A pointer head scores the supplied answer options after an optional reasoning chain. The code fits separate probability temperatures for direct and reasoned decisions.
- A confidence threshold can skip reasoning on easy-looking questions. A chain-length cap controls how long difficult cases take.
- The author reports 0.896 accuracy on 231 public JevBench items for Jeeves v2, versus 0.866 for Jev, with reasoning on 29% of questions. The sealed judge tier is excluded; these results were not reproduced here.
- Jeeves v1 scores 0.913 on those public items. V2 sometimes sounds confident on hard policy and multi-hop questions and skips the reasoning it needs. On knowledge questions, Jev still leads in the author’s table.
The gate has its own failure mode
The README reports about 20 ms for a no-thinking request and 1.2 s median with thinking on one H100 in FP8, but roughly 8 s at p90 for full chains. Those are author-reported settings, not an application-wide latency guarantee. Its non-JevBench comparison with Jev and Kev uses different test items from the same sources, so do not read that row as a paired win.
I inspected the README, pointer-head code and API at revision 8c6ab532b49a578434b3f70062c743e7066a0be0. I did not run the model or reproduce its tests. The training recipe is only partly reproducible: its reasoning chains for supervised training are not published. A typed answer and a confidence score do not establish a correct decision; a confidence gate needs an error audit on the cases it sends down the fast path.