RunSybil tests Jev in four parts of its security-reporting pipeline: filtering weak findings, screening duplicates, classifying synthetic findings and labelling real ones. The real-data failures explain more than the headline speed claims.

Choose how much noise the filter can admit

RunSybil reports a comparison on 117 valid findings and 45 speculative or low-impact findings:

  • Default Jev kept 67 valid findings and blocked 32 invalid ones.
  • Its tuned Jev system kept 74 valid findings and blocked 39 invalid ones.
  • Untuned frontier-model controls kept 27 to 56 valid findings, while blocking 41 to 43 invalid ones.

Tuning used TypeSafe’s autoresearch loop and a CatBoost classifier, a learned combination of Jev’s question scores. RunSybil says the tuned result used an independent holdout. This compares a tuned system with untuned controls, not equally optimized models.

Compared with the frontier-model controls, the tuned filter retained more real findings but also admitted more weak ones. Its threshold makes that tradeoff adjustable.

Screening duplicates is different from proving a match

RunSybil replayed historical decisions to shortlist reports that might share a root cause. A later judge checked the candidate pairs.

The company reports 50.6% precision for Jev, against 44.8% for its current screener. These figures measure agreement with that judge among assessed candidates; they leave missed matches outside the shortlist unresolved.

The source gives conflicting speed summaries for this experiment: its opening says 44× faster, while its table labels the p95 timing improvement 73×. The precision comparison carries a clearer meaning than a single speed multiplier.

Synthetic accuracy hid a real-data problem

On 350 synthetic findings, RunSybil reports Jev got both the OWASP security category and outcome right on 345. On its real-data development set, the initial setup exactly matched only 10%, often choosing the broad “Insecure Design” category.

After changing thresholds and question wording, RunSybil reports 95% exact match on 20 development examples and 10 of 10 on a small holdout. That holdout supports further testing, not a general accuracy guarantee.

Typed answers still need task-specific labels, representative examples and a decision policy. A schema-valid security category can be too broad to help the developer fix anything.

Caveat: all results and performance claims above are RunSybil’s, not independently reproduced. The company describes evaluation work and plans to bring Jev into its live pipeline.