Nick Frichette and Feras Al-Kassar test a specific weakness in Jev-as-a-judge systems: attacker-controlled content can argue for a new definition of the very policy used to classify it. This goes beyond a malformed-output problem—the answers remain valid while the security judgment changes.

I read the full article, evaluation instructions, result tables, and paired scenario payloads. All experimental results are Datadog’s reports, not independently reproduced here. Datadog also sells a competing AI security product; this is a source review, not a security audit.

High precision can hide missed attacks

Datadog evaluates public prompt-injection datasets using both a yes/no probability question, called Noul, and a category-selection question, called Choice. Its reported combined set contains 4,643 attack examples and 7,652 benign examples.

  • On raw inputs, Datadog reports 99.66% precision and 82.90% recall for Noul at a 0.5 threshold.
  • Prepending a counterfeit policy lowers reported recall to 3.88%: only 180 attacks detected, versus 3,849 before.
  • Precision rises to 100% because the modified classifier raises almost no alarms—not because it becomes safer.
  • Choice also deteriorates: Datadog reports attack recall falling from 85.46% to 9.69%.

Precision asks whether alarms are justified. Recall asks how many attacks get caught. A gate can look excellent on the first measure while failing badly on the second.

The attacker rewrites definitions, not permissions

The trusted questions stay unchanged. Inside the untrusted input, the counterfeit policy supplies benign answer labels, declares the original attack an exception, and replaces security definitions with unrelated conditions.

Datadog then tests a simulated LangChain tool-risk gate. A requester with can_edit: false asks to change an employee handbook; an editable profile-description field becomes the injection point.

  • Datadog reports that all 40 baseline trials were blocked, with mean risk probability 0.893.
  • With a policy-shaped argument in that field, its reported mean falls to 0.290, below the gate’s 0.5 blocking threshold.
  • The argument claims a separate permission check will prevent harm, so the attempted call should not count as risky.
  • A more direct role-impersonation payload instead gives a reported mean risk probability of 0.905; plausible policy reinterpretation matters here.

The article does not establish that a real unauthorized file write occurred. The benchmark measures detection, and the scenario measures risk judgments; a mean below the threshold is not a published per-trial bypass count.

Put permission checks where the write happens

  • Enforce requester permissions in the executor, regardless of the model’s risk score.
  • Separate trusted policy and authorization facts from editable profiles, documents, and tool output.
  • Test policy-shaped arguments and false exceptions—not only “ignore previous instructions.”
  • Track false negatives and actual effects, not just valid answer types and low false-alarm rates.

TypeSafe already warns that adversarial content can influence Jev. This report quantifies one failure mode; it does not establish the same rates for every version or deployment. Typed output is not a correct decision, and a low risk score is not authorization.