Red Hat’s AI Safety team tested nine guardrail configurations across two balanced benchmarks: prompt injection and content safety. The result cuts through the category hype. Jev was competitive, but neither “decision model” nor “LLM judge” was a universal winner.

What was compared

  • Purpose-built CPU classifiers: a 200M-parameter DeBERTa injection detector and a 125M-parameter Granite safety model.
  • Older zero-shot classification with BART-large-mnli.
  • LLM judges: Shieldstral 3B, Nemotron 3.5 4B, and Qwen3.6-35B.
  • Jev-style models: Jev 1.13.0, open Laya, and DiffusionGemma through vLLM’s experimental System One endpoint.
  • The labeled, class-balanced datasets were run through EvalHub and NeMo Guardrails; Jev used TypeSafe’s remote API while the other larger models ran on Red Hat’s US-hosted infrastructure.

Prompt injection favored different tools

  • Qwen3.6 led accuracy at 89.31%, followed by the small DeBERTa classifier at 89.01%.
  • Jev reached 86.35% at 348.1 ms median latency.
  • Laya reached 85.44% at 119.3 ms; DiffusionGemma reached 87.72% but was slowest at 561.7 ms.
  • The purpose-built DeBERTa model was fastest at 54.1 ms, while remaining within 0.20 percentage points of the top accuracy.

Content safety favored Jev on accuracy

  • Jev led at 86.20%, with DiffusionGemma at 85.53%, Qwen3.6 at 85.47%, and tuned Nemotron at 85.07%.
  • Jev’s median latency was 360.4 ms; Qwen3.6 was faster at 307.6 ms, and Nemotron was faster still at 241.5 ms.
  • The 125M Granite classifier reached 80.27% but answered in 33.2 ms median.
  • The measurements are Red Hat’s results. I reviewed the published methodology and tables but did not rerun the model calls or infrastructure.

Prompt wording is part of the model

  • Laya initially scored only 57.87% on content safety with the policy written for the other zero-shot systems.
  • A Laya-specific policy raised it to 75.20%, a gain of 17.83 percentage points, while median latency rose from 118.0 to 289.4 ms.
  • Giving that tuned Laya policy to Jev reduced Jev from 86.20% to 82.53%.
  • “Same task, same policy” therefore was not model-neutral. Interface and prompt fit materially changed the comparison.

What the benchmark actually supports

  • Where labeled data and a stable risk definition exist, a small task-specific classifier can be the best operational answer.
  • Jev-style models offer reuse without collecting a new training set for every policy, but did not consistently beat open alternatives or LLM judges.
  • Typed output removes parsing failures; it does not make the block/allow judgment correct.
  • Latency numbers include different network paths, so they are deployment measurements—not pure architecture speed tests.

Caveats

  • The study covers two English guardrail tasks, not decision modeling in general.
  • Jev was remote while local and cluster-hosted models used different hardware and network routes.
  • The authors adapted policies from LLM-judge prompts and acknowledge those may not be optimal for zero-shot decision models.
  • Accuracy is appropriate because the datasets were balanced, but production base rates and false-block costs may differ sharply.
  • Red Hat develops and ships several evaluated guardrail components, so the open configurations and full precision/recall tables matter more than the article’s framing.

“Decision models like Jev do not reliably outperform LLM-as-a-judge, pre-trained predictive models, or open source decision models in speed or accuracy.”

The useful shift is not replacing every judge with Jev. It is treating guardrails as classification systems again: define the error costs, compare against small trained baselines, measure on the actual policy, and choose the least complex model that clears the bar.