rev is an inspectable alternative to calling Jev: it turns a frozen Qwen model into a bounded decision engine. The project includes implementation code, per-item benchmark outputs, and a local endpoint with Jev’s request and response shape. I inspected the code and saved results; I did not run the model or reproduce its benchmark.

How it works

  • On Apple Silicon, the MLX engine renders a state, criterion, and options as a prompt, then reads next-token logits for option letters. It does not generate an explanation or train new weights. The default Qwen3.5-2B is quantized to 8 bits.
  • base.py makes a second pass with reversed options when the first answer is uncertain. It can change the choice, but reported confidence stays at the lower of the two readings for the chosen option.
  • serve.py exposes /v1/systemone locally; remote.py can instead read logits from an already-served Qwen model through sglang. The code falls back to repeated one-token sampling if that server refuses exact log-probabilities.
  • calibrate.py offers per-option bias offsets and temperature fitting on separate labeled examples. A softmax over allowed answer tokens ranks those options; it does not, by itself, establish real-world probability or decision quality.

What the evidence says

  • The author reports 0.550 accuracy on 111 public hard JevBench items for the local 2B model, versus 0.730 for Jev. The README reports 0.784 for a remotely served 27B Qwen run, but the checked-in jevbench-hard-27b.json records 0.775; treat the precise 27B figure as unresolved, not independently verified.
  • The author also reports that a 0.9 confidence gate on the 2B model’s public standard tier answers 47% of questions with 0.941 accuracy on answered questions. That is a result on this sample, not a guarantee for a new workload.
  • The project’s own failure cases include arithmetic/date questions, an inverted email-triage question, and confidently misclassified secrets or vague write requests. It puts coarse secret and vague-action checks in code instead of trusting the model.

The useful distinction: a choice confined to valid labels is not necessarily a correct choice. The public items and prompt/readout choices are visible to the author; the repository says it has not entered the benchmark’s independently run hidden-set leaderboard. Test on your own held-out decisions before using the output as an action gate.