Lidan Hazout at Capsule Security compares Jev with Capsule’s specialized detector for rogue agent actions. An inline detector checks a model’s proposed tool call before the surrounding software executes it.

Capsule reports a substantial advantage on its own held-out test set. The published comparison leaves several details needed to judge that advantage unresolved.

What Capsule reports

These are Capsule’s measurements, without independent reproduction:

  • Accuracy: Jev answers 82% correctly, compared with 97% for Capsule’s detector.
  • Response time: Jev takes 100–500 milliseconds per decision, compared with 78 milliseconds for Capsule’s detector.
  • Training approach: Capsule describes labeling real agent traces, reviewing ambiguous cases and constructing small edits that reverse the correct verdict.
  • Specialization: Capsule uses LoRA, a method for training a small subset of model parameters, to tune its detector to agent-security decisions.

The contrast pairs a general classifier with a detector trained for the evaluator’s own workload. That makes the test relevant to this task while limiting broader conclusions.

Ask which mistakes the detector makes

The article gives no evaluation-set size, class balance, exact Jev version or full prompt. Its aggregate accuracy also leaves the competing rates of blocked harmless actions and missed dangerous actions unclear.

  • Separate false alarms from missed attacks at the threshold the application will actually use.
  • Test closely matched safe and unsafe actions, including instructions buried earlier in a session.
  • Publish prompts, model versions, timing conditions and examples of failures.
  • Check the proposed action against permissions and deterministic rules before execution.

Capsule also cites a public StepShield result for its own detector. The article supplies no matched Jev result on that benchmark, so it cannot independently validate the reported head-to-head gap.

Update the context comparison

The article describes Jev’s default context limits as its ceiling. TypeSafe’s current documentation supports an opt-in 262k-token limit for both the request and each question.

TypeSafe warns that this mode is slower and less accurate than the defaults. The default limits remain 64k tokens per request and 32k per question, including the shared state. A larger advertised window still needs testing on long, noisy and adversarial sessions.

Keep correctness separate from valid output

A typed allow/block answer gives application code a predictable value to handle. Correctness depends on the evidence, policy, operating threshold and behavior under attack.

Capsule’s test motivates checking a specialist baseline. Choosing a production gate still requires error breakdowns and workload-specific tests.

Verification: read the complete article and checked its context claims against current TypeSafe documentation. No detector execution or live performance reproduction was performed.