Artificial Analysis launched a benchmark for models that choose among supplied answers. Its launch results put Jev ahead of other decision models overall, with substantial differences between tasks.

All figures below are Artificial Analysis’s reported measurements; they were not reproduced here.

Read the component scores before the ranking

The Decision Index weights four tests equally: factual knowledge and abstention, difficult multiple-choice questions, tool-use choices, and safety classification.

  • Composite: Jev 1.13 scores 58, ahead of GLiDE and Microsoft-Decision-1 at 54. This is a weighted index, not general accuracy.
  • Reasoning: Jev scores 21 on HLE-MultiChoice, versus 15 or less for the other decision models in the launch comparison.
  • Tool use: Jev scores 81 on When2Call; Celeris-1 Decision and GLiDE lead at 84.
  • Safety classification: Jev scores 94, behind GLiDE at 95. The metric averages three F1 scores, balancing missed cases against false alarms.

The methodology narrows the task set. When2Call excludes ambiguous clarification cases, and the safety subset selects unanimous human labels while excluding questions that consistently trigger model refusals.

Confidence depends on the question

Artificial Analysis finds every decision model overconfident overall. Knowledge and reasoning drive much of that gap; tool-use and safety classification are better calibrated.

On its knowledge test, Jev scores 35 for accuracy and 35% for the non-hallucination measure. The latter counts “I don’t know” among questions the model failed to answer correctly. It is not the share of all answers that are trustworthy.

Typed output is not a correct decision. Test confidence against labelled examples from the actual workflow before choosing an automation threshold.

Keep benchmark cost separate from application cost

Artificial Analysis reports $42 per million benchmark tasks and 0.13 seconds median time per task for Jev. These figures describe this evaluation’s input mix and measurement conditions.

Its comparison with Claude Haiku 5.5 (Low) reports a higher index score of 61, alongside $184 per million tasks and 1.4 seconds per task. That is a measured capability–cost tradeoff within the suite.

A benchmark task bill excludes the surrounding application: retrieval, retries, stronger-model fallbacks, human review, and hosting. Median response time also leaves slow-tail behaviour unresolved.

Caveat: these are the evaluator’s launch-snapshot results, with filtered test subsets and partly private data; local reproduction and workflow-level validation remain open.