Shrivu Shankar at Abnormal tested Jev, OpenAI’s Decisions API, self-hosted open decision models, and frontier LLMs on synthetic email-security cases modeled after difficult production examples. This is a vendor’s internal evaluation, not an independently reproduced public benchmark; its real customer emails and full evaluation set are not available for inspection.

The decision that matters

  • The models answer whether an email is an attack, classify its broad category, and identify an attack type. Returning a typed probability is useful, but it does not establish that a message is safe or malicious.
  • Removing a legitimate email from an inbox is costly, so the relevant measure is attacks caught while holding false positives very low—not just overall accuracy or an aggregate score.
  • Abnormal says Jev was the best calibrated model in its comparison, but the OpenAI Decisions API caught more attacks at its high-precision operating point. Those findings belong to this particular synthetic dataset and threshold policy.

Fine-tuning changed the tradeoff

  • Abnormal reports that a Kev-4B fine-tune raised attack recall at a nominal 99% precision from 7% to 66%; I did not reproduce the training or results.
  • The same run reportedly worsened attack-type macro-F1 from 68 to 37. Improving the trained decision damaged a different question answered by the same model.
  • Its claimed lead over a frontier model has overlapping confidence intervals, and thresholds were first chosen on the reported test split. The authors also provide a separate threshold check; neither setup mirrors real-world attack prevalence.

Serving is not a benchmark chart

  • Cost comparisons assume highly utilized GPUs; Abnormal notes that real email traffic peaks and regional data rules make an always-on fleet harder and costlier.
  • Their proposed deployment is layered: cheap prefilters first, decision models for selected traffic, and more expensive reasoning only for ambiguous cases.

The practical lesson is to evaluate the action and its cost, choose thresholds on separate data, and retest each task after fine-tuning. A legal typed response is not necessarily a correct security decision.