stern9’s jev-bench gives Jev and Claude the same typed questions, then measures correctness, client-observed latency, probability calibration, and estimated token charges. Its value is a readable evaluation scaffold with explicit mistakes—not a universal model ranking.

I inspected the README, task loading, provider calls, answer parsing, metrics, report CLI, smoke tests, and published results at cbe3a57c3cb7b9f24fdad53c97a208981046ba83. I did not run the harness, its tests, or either provider. This was source inspection, not a security audit.

Three tasks, different failure boundaries

The maintainer’s dated report covers 72 hand-labeled examples: support-team routing, phishing checks, and incident severity.

  • The README reports 94.4% accuracy for Jev, 91.7% for Haiku 4.5, and 98.6% for Opus 5.
  • Every model passes the report’s 24 phishing examples; that small clean result is not a phishing-safety guarantee.
  • Jev’s reported severity accuracy is 86.4%, with mistakes between adjacent severity levels.
  • A mixed login/browser ticket goes to the wrong team with recorded prediction probability 0.79.

These are maintainer-recorded results, not independently reproduced. The checkout contains the report and labels but no raw records.jsonl from the published run, so its historical provider results cannot be replayed from the committed artifacts alone.

Calibration is measured, not granted by the schema

  • Choice confidence uses the selected answer’s probability; score questions use the most probable level rather than a rounded average score.
  • Noul answers become yes/no decisions at a probability threshold of 0.5.
  • The harness computes five-bin expected calibration error—how far confidence differs from observed correctness—and Brier score, which evaluates the full probability distribution.
  • Claude’s probabilities come through a structured-output adapter; they are not the same kind of measurement as Jev’s model probabilities.

The report gives Jev a routing calibration error of 0.025 on 26 examples. That is the maintainer’s result on a tiny task, not a reliable production threshold or an independent calibration validation. Typed output is not a correct decision.

Reuse the harness, tighten the protocol

The implementation makes real TypeSafe system_one() calls and routes Claude through system-one-adapter. Identical question definitions help comparability, but the README’s “fair by construction” language is too strong: the providers perform different work, and Claude uses adapter-managed structured outputs and retries.

  • Pin a Jev version instead of the moving jev-latest alias used in the report.
  • Preserve raw records, exact provider settings, and repeated trials before treating small differences as stable.
  • Add your own held-out, disputed, and adversarial cases; severity labels encode an operational policy, not universal truth.
  • Keep failures visible: accuracy counts failed calls as wrong, while latency, calibration, and cost summaries exclude error records.

The maintainer reports per-task Jev median latencies of 175–193 ms and estimated charges of $0.0154–$0.0185 per thousand calls. Neither was reproduced here; latency includes network time, and costs use hard-coded token rates rather than invoices or total workflow cost.