Mateus Fonsek’s decisions-bench is a useful, inspectable comparison of Jev, OpenAI’s Decisions API, and the locally run Laya model. Unlike a launch claim, it publishes Portuguese test cases, adapters, response logs, and a reporting pipeline. I inspected the repository and recorded results; I did not rerun the paid API calls or the local model.

Three distinct decisions

  • Agent routing: In the author’s 150-message set, Jev selected the labeled tool in 88.0% of cases versus 86.7% for OpenAI and 30.7% for unrotated Laya. Both hosted models reached only 80% on the ambiguous subset.
  • Prompt-injection screening: On 75 attacks and 75 benign texts, Jev’s author-reported F1 was 96.6% versus 95.8% for OpenAI. Jev flagged 4% of hard benign examples that merely discussed attacks; OpenAI flagged none in that subset but missed more attacks.
  • Answer grading: Jev matched the author’s one-to-five grade exactly 80.7% of the time versus 72.0% for OpenAI. OpenAI’s grades were within one point every time in this set; Jev’s were not.

The local-model lesson

  • Laya was fast on the test machine, but its routing predictions concentrated on a single option. Rotating all eight option positions and averaging lifted the author’s measured accuracy from 30.7% to 40.0%—at the cost of eight calls per decision and much of the latency advantage.
  • The author reports hosted median end-to-end latency around 213–229 ms depending on task/provider. Subtracting an unauthenticated request’s round-trip time to estimate model time is only a rough proxy, not direct server-side timing.
  • Published per-call prices are estimates from logged tokens and a checked-in price table, not bills independently audited here.

What the comparison cannot establish

The 450 cases were generated by Claude and only quickly reviewed by a human; they are synthetic labels, not independent production ground truth. The runs are zero-shot and one pass per case. A cleanly typed probability is not proof that a decision is right or calibrated on your data.

The useful artifact is the replayable evaluation design, especially its hard negatives, per-option response logs, and explicit distinction between routing accuracy, attack recall, and grading error. The repository’s results were inspected, not reproduced.