Vals AI tested TypeSafe’s Jev against other models on bounded decisions, rather than asking it to generate prose. Its two tasks give sharply different answers to the question of whether the cheaper classifier can replace an LLM.

Where it held up

  • On 400 claim-verification cases built from company filings, Vals reports Jev got 390 right, tying GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna. The labels were fixed by construction and the cases human-audited; these are Vals’s measurements, not results I reproduced.
  • Vals reports roughly $0.02 per 1,000 cases for Jev versus $0.08 for Luna and much more for frontier models. More judgments against the same document added little Jev latency in its test.
  • A bounded answer shape prevents malformed labels. It does not ensure that the selected label is true.

Where it failed

  • On a preregistered 396-question, 12-subtask LegalBench slice, Jev ranked last of the twelve tested systems: 0.730 class-balanced accuracy. It answered “Yes” on all 33 contract_nli_permissible_copy questions, although the answers were nearly evenly split.
  • Calibration did not transfer: Vals measured Jev’s expected calibration error at 0.011 on claim verification and 0.108 on LegalBench. A probability useful on one workload is not a universal safety threshold.
  • Vals tuned an automation threshold on the first 30% of claim cases for a 1% error budget. On the held-out 70%, Jev automated 95% but incurred 1.6% error. Six of twelve systems overshot the same budget.

The filing task uses constructed edits, while LegalBench is a small, public slice that may overlap LLM training data. Neither is a substitute for testing real cases from your own workflow. The useful lesson is methodological: compare paired labels and costs, then freeze thresholds before measuring held-out errors.