Chris Mungall’s Jevotron (jt) is a CLI for finding suspicious fields in CSV, JSON, YAML, text, SQLite, DuckDB, and other structured data. It is more than an API wrapper: it handles parsing, field selection, request previews, resumable caching, local filtering, and exportable review queues.

What the tool adds

  • preview exposes the exact Jev request without using an API key or making a model call.
  • scan assesses entries independently, retains field probabilities and source locations, and streams successful results into SQLite.
  • Request hashes let unchanged entries reuse prior assessments after files are reordered or reporting thresholds change.
  • DuckDB-backed filters can shortlist rows or fields by label, confidence, and other local report columns.
  • Custom Python parsers and configs cover formats or field-selection rules outside the built-in defaults.

The measured agent-trace pilot

  • The repository includes scripts, manifests, saved predictions, and aggregate results for 24 public AgentProcessBench traces containing 163 assistant steps.
  • Jevotron’s recorded jev-1.13.0 run matched 130 of 163 step labels (79.8%), versus a 71.8% majority-label baseline.
  • For harmful steps, the recorded run reached 89.7% precision and 70.3% recall; final-result labels matched 19 of 24 traces (79.2%).
  • Performance varied sharply: GAIA step accuracy was 68.0%, below its 76.0% majority baseline, while tau2 reached 85.1% step accuracy.
  • These are the maintainer’s checked-in results. I verified the result files and evaluation code, but did not rerun the paid Jev calls.

What I ran and inspected

  • I inspected the TypeSafe HTTP adapter, request construction, cache/report paths, analysis script, public manifests, and saved result tables.
  • I ran the repository’s local test suite from a clean downloaded checkout: 519 tests passed.
  • The tests use fake responses and do not validate Jev’s live accuracy, latency, calibration, or availability.
  • This was an implementation review, not a security audit.

The important limits

  • The pilot deliberately selected six traces per domain, balanced successful and unsuccessful outcomes, and excluded long traces; it is not the full benchmark.
  • Steps inside one trace are correlated, so 163 labeled steps are not 163 independent trials.
  • The model saw the entire completed trace. This tests retrospective review, not online detection before an agent takes an action.
  • Jev returned typed labels and probabilities, but the study measured label agreement—not probability calibration.
  • Neutral versus useful steps were a weak spot, and plausible final answers sometimes hid earlier unfulfilled requests.

The useful pattern is not “trust the score.” It is use Jev to build a focused review queue, retain the evidence and probabilities, and inspect both flagged cases and a sample of apparently clean ones. Jevotron makes that operational loop concrete and auditable.