Harness engineer Ritabrata Moitra describes replacing the yes/no decision inside an AI-API discovery pipeline. The application still needs four stable fields—two flags, confidence and a human-readable reason—and an LLM still enriches positive findings. Jev is not a replacement for the whole pipeline.

The comparison

  • Harness sampled representative API request/response spans, trimmed them and evaluated 500 stratified endpoints; labels were seeded by its single-shot LLM pipeline and checked by staff, especially ambiguous cases. This is an internal study, not an independent reproduction.
  • On the is_ai_api flag, Harness reports 98.6% accuracy for Jev, with four false positives and three false negatives; the single-shot LLM was about 99% accurate, with higher reported precision and recall.
  • Harness estimates $176 per thousand endpoints for its older summarize-then-classify workflow, about $22 for a single LLM call, and $0.14 for Jev plus narration on positives. The cheap comparison changes both preprocessing and model; do not attribute the entire saving to Jev alone.
  • Its measured Jev decision latency was 0.80 s median / 1.11 s p95, versus roughly 16 s / 28 s for the single-shot LLM. These are the team’s measurements on its workload; I did not rerun them.

The useful control surface

  • All seven Jev errors in Harness’s sample fell in its 0.3–0.8 probability band, which covered 17 endpoints. Harness plans an LLM fallback for uncertain scores, rather than treating every typed answer as correct.
  • Positives get their reason from the existing LLM enrichment call; negatives get a template. This preserves downstream field shape while keeping generation off most decisions.
  • The article says it is deploying a 0.2–0.8 fallback, but quotes a call estimate from a 0.3–0.8 band; the cost table should not be read as a validated end-to-end cost of the final fallback policy. Nor do zero observed errors at the score extremes prove perfect calibration on future tenants.

For a security inventory, three missed AI endpoints are not harmless just because overall accuracy looks high. The transferable design is a stable contract, reviewable scores and a measured escalation path. Source analyzed; no paid calls or deployment reproduced. Related: Cisco’s safety-classifier comparison tests decision models against a specialized classifier on a different task.