Cisco researchers Prashanth Arun and Jin Ma compare hosted Jev and open-weight Laya with Cisco’s own fine-tuned encoder on content-safety policy categories. The question is not whether models can return a well-formed probability, but whether that score catches harmful content without falsely flagging too much safe content.

What they tested

  • Cisco used 24 harm categories and six datasets, applying one yes/no question per category and perspective to Jev and Laya. Its custom 1B-parameter encoder was trained directly on the policy taxonomy.
  • At a matched 0.5% false-positive rate, Cisco’s trained model had the best recall on all six datasets. On its multi-turn set it reports 61% recall, versus 34% for Jev and almost none for Laya. These are Cisco’s measurements; I did not reproduce them.
  • Against a prompted Gemma 4 31B judge on eight public benchmarks, Jev had higher full-curve AUC on seven and tied on one, but Gemma led at the strict false-positive budget on several datasets. AUC across all thresholds and usefulness at a specific threshold are different questions.

What the numbers cannot establish

  • Cisco says the six-set reference labels were generated by an LLM, not human-adjudicated. Errors in those labels affect every reported comparison.
  • Thresholds were picked separately for each model and dataset to maximize recall inside the budget. Those are analysis points, not preselected deployment settings; Cisco calls for separate validation and held-out human labels.
  • A specialist trained to this taxonomy has an inherent task-fit advantage. Conversely, Jev can address a new category without training; the article does not show that its probabilities are calibrated for a new organization’s production policy.

The operational lesson is to test at the false-positive budget your team can afford, then lock thresholds on separate data. Typed outputs make the wiring cleaner; they cannot guarantee the safety decision. Article reviewed; models not run. Related: Harness’s AI-API classifier shows a different task where an uncertainty fallback is central.