Michael Stalcup tests Jev on three tasks at CaseTrail, a forensic-accounting software company. The article reports useful results and explains why the team still refuses to send client records to the service.

All measurements below are CaseTrail’s reports from synthetic data; they were not reproduced here.

Two tasks improve; difficult name pairs get worse

CaseTrail uses known-answer datasets: 600 transactions, 174 entity pairs, and 84 pages spanning 28 document types.

  • Transaction tagging: Jev reaches 97.5% accuracy, versus 96.2% for GPT-4o mini.
  • Entity matching: Jev reaches 93.7%, versus 98.9% for GPT-4o. Deliberately confusing look-alikes are its weak spot.
  • Page classification from text: Jev reaches 98.8%, versus 97.6% for GPT-4o mini. Both routes need text extraction when pages are scans.

The same name can have aliases, and different people can share a surname. Those judgments need a separate test from recognising a document type.

Confidence can route work, but needs its own evaluation

CaseTrail reports 99.4–100% correctness among Jev’s high-confidence text decisions, using a 90% confidence threshold. On entity matching, that threshold accepts only 45% of pairs.

Its Jev-first cascade sends the remainder to GPT-4o and reportedly matches GPT-4o’s 98.9% accuracy at 59% of its cost.

High-confidence precision measures the accepted subset. It does not establish that every probability is calibrated, or that the threshold will transfer to client documents. Typed output is not a correct decision.

Compare the complete request and the complete route

  • CaseTrail reports single-item median times of 0.07–0.08 seconds for Jev, versus 0.85–1.05 seconds for its comparison models.
  • The OpenAI timings include a hop through CaseTrail’s own API, so this is a comparison of those application routes.
  • Transaction tagging costs reportedly converge: $2.00 per 100,000 items for Jev and $2.18 for GPT-4o mini.
  • Repeating the answer options increases Jev’s input volume. A cheaper input-token price need not produce a much cheaper decision.

Data terms and images block adoption

CaseTrail says litigation records can carry protective orders restricting who processes them. It wants written retention terms and a data-processing agreement before considering production use.

The article also reports that 43% of a separate 1,000-file customer sample lacks built-in text. Its synthetic image trial looks promising, but CaseTrail describes that Jev feature as experimental at test time.

Caveat: the test uses synthetic examples, small task-specific datasets, and application-specific routes; the reported threshold, timing, and cost need validation on an authorised production workload.