iammrduncan’s inference-benchmarks saves the evidence behind its decision-model scores. Each run records the model, serving software, request settings, raw responses, and scoring rules. Its Jev comparison can be recalculated without calling a model.

Replaying results is different from rerunning inference

Local checks rebuilt four committed summaries byte for byte, including Jev’s full and quick tests and Qwen’s quick test. The runner’s 22 offline tests passed. Those checks reproduce the scoring of the author’s saved responses; they leave the original model calls, timing, and billing unverified.

The author’s full Jev run contains 400 cases and 2,000 decisions across support, invoices, security incidents, and agent traces. It records 73.8% accuracy and 154 ms median per case request. Each request answers five questions, so that timing must stay separate from per-decision cost.

Compare the same cases—and the actual request shape

On the author’s smaller, matched 100-case, 500-decision subset:

  • Jev: 74.8% accuracy, 171 ms median per case request, and an estimated $0.00745 per 1,000 decisions.
  • Qwen 3.8 27B on Cerebras: 72.2% accuracy, 572 ms median per case request, and an estimated $0.606 per 1,000 decisions.
  • Qwen goes through a gateway that makes five model calls per case; Jev answers the whole case through its native API.
  • One Qwen case fails output validation. Its five decisions count as wrong, while probability-quality metrics use valid answers only.

The accuracy intervals overlap. These figures describe the author’s two implementations on one public test, rather than a general ranking of the models.

Score probability quality separately

The scorer checks answer distributions, counts missing answers as errors, and measures how closely probabilities match the reference distributions. Typed output is not a correct decision.

Its uncertainty estimates treat the five related decisions within each case as independent. The code explicitly warns that this makes the intervals too narrow. Its confidence-calibration metric also uses a definition that cannot be compared directly with the dataset card’s figure.

Keep the accounting visible

Costs come from recorded token usage and a saved price table. They exclude running the gateway, application hosting, and human review; they are estimates rather than independently checked invoices. The Jev run also records a dirty source checkout, which weakens exact reconstruction of the original execution environment.

Caveat: implementation inspected and offline scoring exercised; no fresh model inference or security audit was performed.