AI Profs built a unusually inspectable classroom benchmark: 18 fictional business cases, seven documents per case, bounded questions for Jev and Claude Opus 4.8, and the same visible policy code converting both models’ answers into actions.

The latency result is substantial. The quality headline is much less settled because one author wrote the cases and reference answers, faculty had not reviewed them, and five repeated runs of each case are not independent examples.

What the experiment held constant

  • Six decision families cover market entry, investment, product launches, suppliers, consulting diagnosis, and fictional credit review.
  • Each case contains about 1,000 words across seven documents, including distracting details, contradictory claims, and later corrections.
  • Both models answered the same bounded questions; the same code and thresholds produced the final decision.
  • Each case ran five times per model, for 90 measured runs each, plus one warm-up request.
  • Jev used one API call for all questions. Opus returned probabilities for every option through Claude Code at low effort.

Jev was much faster and cheaper in this harness

  • Median decision latency was 315 ms for Jev versus 6.46 seconds for Opus—about 21× faster, according to AI Profs.
  • Projected cost per 1,000 decisions was $0.097 versus $12.19, or about 125× cheaper at the published prices used on October 3.
  • Jev’s p95 was 453 ms; Opus’s was 8.38 seconds. One Opus request took 159.3 seconds.
  • These are the project’s measurements, not results I reproduced. Opus timing excludes CLI startup, which favors Opus.

The accuracy headline needs restraint

  • Final decisions matched the author’s reference on 90% of Jev runs and 84% of Opus runs.
  • The models’ most frequent decision agreed in 17 of 18 cases.
  • Both reached 97% accuracy on yes/no answers.
  • Opus had the lower Brier score—0.022 versus Jev’s 0.030—so Jev’s higher final-decision agreement does not mean it produced uniformly better probabilities.
  • Some scored questions remained weak: Jev hit only 7% on investment customer concentration and 33% on optimistic repayment assumptions.

A typed probability is still a model judgment. The benchmark makes those errors visible; it does not turn them into ground truth.

The best finding was a bug in the policy

An early market-entry rule permitted a pilot only after demand was already proven. Both models exposed the backward logic: uncertainty is precisely why a company pilots.

Because the policy lived in code, the authors changed one line and replayed stored model answers without paying for new calls. That separation—model judgment first, explicit organizational rule second—is more convincing than the model-versus-model leaderboard.

What this does and does not show

  • It shows that Jev can process long, messy synthetic documents and feed stable typed answers into ordinary policy code at low latency.
  • It shows the harness, cases, per-question results, latency distribution, token counts, and caveats rather than only a vendor headline.
  • It does not establish superiority on real credit, fraud, investment, or operational decisions.
  • The reference labels came from the project author, not domain experts; two model disagreements prompted the author to reconsider the reference itself.
  • Five runs per case test repeatability, but they do not create 90 independent cases.

“The most instructive mistake was ours.”

The practical result is not “Jev beats Opus.” It is that bounded model judgments plus versioned policy code produce a system whose speed, uncertainty, rules, and mistakes can be inspected separately.