Tobias Deußer, Lorenz Sparrenberg, and Rafet Sifa evaluated the pinned jev-1.13.0 model on public datasets and released their harness and raw responses. This is a measured independent study, not TypeSafe’s launch benchmark. I read the paper and inspected its reported methods and results; I did not rerun its requests or independently verify its released response archive.

What they actually tested

  • One frozen, zero-shot question template per dataset, selected after a small development pilot; the evaluation covers 37 datasets and 346,009 answered requests.
  • Two open-weight reference models, Qwen3.8-27B and Gemma-4-E4B, received the same state and questions. Their next-token probabilities over answer codes were scored without generating reasoning text.
  • The authors report Jev ahead of Qwen on 27 datasets, behind on nine, and tied on one; it led Gemma on all 37. This is not a comparison against thinking-enabled frontier models or a proof that Jev is generally more intelligent.
  • Their reported run cost $9.15 and averaged 0.36 seconds of client-side latency per request under concurrent load. Those are their measurements at the stated version and pricing, not measurements reproduced here.

Calibration depends on the question shape

  • Choice answers were reasonably calibrated in aggregate: the researchers report expected calibration error (ECE)—the gap between stated confidence and observed accuracy—of 0.028 across Choice answers. That average masks weaker tasks and languages.
  • Binary noul questions tended to assign positives too little probability in single-label tasks, but too much across sparse multi-label tasks. Typed probabilities do not make a default 0.5 decision threshold safe.
  • On the UNFAIR-ToS legal-clause dataset, tuning thresholds on separate development examples raised Jev’s reported micro-F1 from 0.499 to 0.748 on the evaluation split. It did not fix the harder German legal-judgment dataset, where F1 stayed at 0.204.
  • Rejecting uncertain decisions also helped: on Banking77 intent routing, reported accuracy rose from 79.7% overall to 96.3% on the most confident half. The other half still needs a fallback, and coverage matters.

The limits worth noticing

  • Jev’s reported performance fell on low-resource languages, fine-grained emotions, legal judgments, and ratings of generated text. On one prompt-injection set it had perfect precision but only 50% recall at the default threshold—a risky tradeoff for a gate.
  • Its unusually high accuracy on calculation-heavy MMLU questions raises a possible training-exposure concern. Rotating answers and hiding the question ruled out simple answer-position shortcuts, not memorized question–answer pairs. The authors do not claim to have proved contamination.
  • Each request was run once; alternate prompt wording and reasoning-enabled baselines might change the comparison. The result applies to jev-1.13.0, not every future alias.

The practical lesson is to measure the decision on labeled examples from your own workflow, set thresholds and an abstention route, and check calibration per task. A correctly typed response is only a well-formed answer—not necessarily a correct one.