JYeswak/jev_playground treats Jev as one candidate technique, not the default answer. Its reusable contribution is an evaluation discipline: freeze labels, compare with the majority answer and a cheap rule, set the acceptance bar before calling the model, and turn off interventions that do not earn their cost.

I inspected the README, TypeSafe SDK client, uncertainty sampler, and review/gate receipts at revision 5e0ef7b09940033f8b7a8346109cf970abc21cc2. The offline gate-bound calculation ran successfully and its 14 acceptance tests passed. I did not call Jev, reproduce live model performance, or audit the system’s security.

Negative results are the useful part

The maintainer’s recorded jev-1.13.0 experiments report:

  • Finding novelty: Jev classified 49/85 review findings correctly, versus 77/85 for a same-issue/class rule and 69/85 for always predicting “repeat.”
  • Verdict survival: Jev accuracy was 0.400 on 55 adjudicated rows, versus 0.618 for both the majority baseline and a keyword rule.
  • The novelty receipt reports all new findings caught—but many repeated findings mislabeled new. Catching everything is not the same as useful precision.

These are author-reported bounded-corpus results, not independently reproduced Jev evaluations. The verdict set has no held-out partition; neither receipt supports a universal claim that rules beat models.

“Caught every harm” needs a denominator

The gate receipt’s 47/47 result concerns harms in the already Jev-flagged sampling group. It is not gate-wide recall. The historical labels came from local LLM labelers, not independent human review, and the model saw a bounded command projection rather than every full command.

My offline replay reproduced the repository’s approximately 0.680 lower recall bound, calculated from its recorded rows and the uncertainty left by 0/200 labeled harms in the random-unflagged sample. This checks the stored-data calculation—not the labels, current model behavior, or production safety. The script correctly reports DESCRIPTIVE_ONLY and zero live calls.

Build a boundary around the score

  • The inspected client pins Jev by default, uses the official SDK, and distinguishes configuration failures from missing or malformed answers.
  • Uncertainty sampling pairs near-threshold examples with a seeded random audit; inspecting only borderline scores would miss confident errors.
  • The README marks memory scoring and skill-veto enforcement as off, and distinguishes copied integration files from proof that extensions actually fired.
  • Fake fixtures and offline replays are labeled as such; a typed response or successful installer does not establish a correct decision.

Typed output is not a correct decision. Use these receipts as a model for deciding what not to ship, then collect representative human labels and measure the end-to-end intervention before promoting a gate.