Jared Palmer’s Kev is not another wrapper around TypeSafe’s API. It is an Apache-2.0 family of Jev-style decision models built on Qwen3.5 and Qwen3.8, with downloadable weights, a compatible /v1/systemone endpoint, training and evaluation code, and model cards detailed enough to show where the headline breaks.

I inspected the README, pointer-head model, API adapter, calibration and benchmark code, the Kev-4B model card, and the published suite layout. I did not download the multi-gigabyte weights, run the models, or reproduce the benchmark results.

The implementation scores options instead of generating prose

  • Kev reuses a causal Qwen backbone but does not use its vocabulary head to generate an answer.
  • Each option is delimited in the input; a learned pointer head compares the question’s final hidden state with each option’s closing-token state, then applies softmax.
  • Questions share a cached state while remaining isolated from one another. Hybrid Qwen3.5 backbones run each question as a separate causal row; attention-only models can use a block-causal packed form.
  • noul becomes a two-option false/true decision, choice maps probabilities back to named options, and score returns the expected index over ordered levels.
  • The server mirrors TypeSafe’s public request and response shapes, so the official SDK can target a local Kev endpoint.

Four sizes, with different boundaries

  • Kev 1.0 publishes 0.8B, 4B, 9B, and 27B variants. The project recommends 4B as the default and describes 27B as an 80 GB GPU model.
  • The smaller checkpoints are LoRA adapters over frozen Qwen3.5 bases; Kev-27B fine-tunes full Qwen3.8 weights.
  • The API serves states up to 65,536 tokens, but the Kev-4B model card validates accuracy only to 8,192 tokens. Serving capacity is not evidence that quality survives to the maximum context.
  • The model card says the 4B server is suitable for a 24 GB GPU or 32 GB Apple Silicon Mac; those are maintainer claims, not measurements I reproduced.

The evaluation story is stronger than the leaderboard pitch

  • Kev-4B’s locked out-of-domain test reports 83.8% accuracy and a 0.224 Brier score over 656 questions. The model card says the test was read once after selection rules were fixed.
  • On the broader 14-dataset breadth-v1 test, Kev-4B reports 69.0% accuracy and a chance-corrected index of 38.0, well below the separately measured Jev index of 54.0.
  • Kev-27B reports 85.1% on the new-source development split versus 85.7% for Jev, but Jev was not run on Kev’s locked test and its unknown training data prevents a controlled architecture comparison.
  • The benchmark code checks option keys, finite probabilities, normalization, rejected records, and grouped variants. It reports calibration, Brier score, selective coverage, confident errors, permutation sensitivity, and latency rather than collapsing everything into accuracy.

Calibration is treated as workload policy

  • Each checkpoint ships with one fitted temperature; that changes probability sharpness but not the winning option.
  • The calibration tool compares raw, shipped, in-sample workload, and group-disjoint out-of-fold temperatures, then bootstraps changes in ECE, Brier score, coverage at a 5% error budget, and risk-coverage area.
  • The code comments warn that the shipped temperature does not necessarily transfer. The project recommends fitting thresholds on labeled examples from the deployment workload.
  • That distinction matters: typed output guarantees a legal value, while calibrated confidence—if it transfers—only helps decide which legal values to automate.

The limitations are part of the result

  • Kev reports weaker knowledge performance than Jev, brittle date arithmetic, option-order sensitivity, and accuracy loss beyond trained context lengths.
  • The 4B model card says some developer-tooling labels may teach dataset heuristics rather than the underlying decision.
  • A single temperature cannot reorder errors; Kev’s reported share of new-source decisions accepted within a 5% error budget remains below Jev’s.
  • The repository’s numbers are unusually inspectable, but they are still the maintainer’s results. I did not run the weights or audit the training data and code for security.

“Calibrated probabilities can create unwarranted trust.”

Kev’s main contribution is not that an open model has “beaten Jev.” It is a reproducible decision-model stack where the architecture, data recipe, held-out tests, confidence policy, and failures can all be examined—and where the authors leave enough evidence to see that a valid typed decision can still be wrong.