Cho Yin Yong’s DecideBench v2 asks decision models to choose an action under a written policy. Each paired case changes one detail that should flip the answer. The repo includes the items, results, model adapters and scoring code.
A harder test than a neat output
- The author’s v2 has 450 cases in 225 contrastive pairs across nine task families. Inputs include raw email, release metrics and retrieved evidence. Claude generated the cases; three Claude models checked the labels, and the author flags one arguable label.
- In the author’s recorded runs, Jev 1.14 scored 88.0% on individual items but 76.4% when both sides of a pair had to be right. Claude Opus 4.8, the reference, scored 99.3% and 98.7% respectively. These results were not rerun here.
- On staged-release rules that require a computed rate or ratio, the author reports 58% for Jev versus 100% for Opus. A typed choice can be valid JSON and still be the wrong operational decision.
- The reported $75.04 per million Jev tasks is API spend under this benchmark’s prompts and list price, not a general user-task bill. Hosted latency includes the network; self-hosted latency does not.
The dataset validates locally and its report regenerates from stored results. The repository’s test run had 233 passing tests and one failing numerical regression pin at the inspected revision. I inspected the dataset, result and scoring code; I did not call the models, reproduce the benchmark or audit the implementation. Build task-specific labeled pairs and retain a fallback before using model confidence to authorize actions.