Open-Jev is an independent, open implementation of the Jev-style decision-model interface. Its new 27B v1.1 checkpoint pairs a pinned Qwen3.8-27B base with LoRA adapters, a scalar decision head, saved calibration temperature, public training code, released data, and unusually extensive evaluation artifacts.

I inspected the repository’s model, training, serving, benchmark, audit, and provenance files. I compiled the Python source and ran the offline unit suite; 1,456 tests passed, 90 skipped, one failed, and one errored, so I am not calling the checkout’s suite a clean pass. I did not download the 27B weights or rerun GPU inference, and no security audit was performed.

The public JevBench result

On the repository’s historical 231-task public subset, the project reports:

  • Open-Jev 2B: 150/231 (64.94%);
  • Open-Jev 9B: 179/231 (77.49%);
  • Open-Jev 27B v1.1: 197/231 (85.28%);
  • TypeSafe Jev 1.13.0: 200/231 (86.58%).

On the 111 “Hard” public tasks, 27B recorded 80 correct, versus Jev’s 81. These are the project’s audited measurements, not results I independently reproduced.

The comparison is narrower than the headline:

  • It covers 231 public tasks, not the complete 534-task benchmark.
  • The adapters used different candidate orders on 119 of 139 Choice tasks.
  • Open-Jev ran locally while Jev used hosted HTTPS, so timing is not a matched-hardware comparison.
  • A fresh independent 27B result on JevBench v1.5.4 is still pending.

What the model actually is

The implementation does not generate an answer string and then parse JSON. It scores each candidate independently:

  • The backbone is Qwen3.8-27B at a pinned revision.
  • A scalar decision head is initialized from the pretrained “Yes minus No” output direction.
  • LoRA rank 8 adapts attention projections while the base model remains frozen.
  • Candidate logits become a normalized probability distribution.
  • A saved temperature calibrates the released checkpoint.
  • Noul is represented as a two-logit decision; Choice and Score receive one score per candidate.

The model package contains adapters, the trained head, and calibration—not merged Qwen base weights. Users still need the pinned upstream base model and Open-Jev loader.

The project takes provenance seriously

The 27B v1.1 report binds the result to:

  • Qwen3.8-27B revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0;
  • checkpoint tree SHA-256 c49994563c3c4f04a99d9130203c4e526f4ae5086c84deec57698d18cb652e71;
  • saved temperature 2.5343690298472983;
  • a predeclared final checkpoint after 37,160 optimizer steps;
  • fixed data manifests, request settings, and evaluation code identities.

The full internal audit accounts for 127,787 expanded held-out records, with exactly one successful prediction per record and no missing or duplicate rows. Its replay recomputed probabilities and aggregates from saved logits.

That is good artifact discipline. It does not make the internal distribution representative of every real workflow.

Internal results are not external generalization

The project reports the 27B checkpoint at:

  • 98.31% on the old internal hard Test rows;
  • 95.98% on old internal hard OOD rows;
  • 96.65% on expanded internal hard Test rows;
  • 96.44% on expanded internal hard OOD rows.

Those panels overlap, and the repository explicitly says not to add their denominators. The 2B, 9B, and 27B models also differ in scale, training history, and data, so the table cannot isolate a pure scale or data effect.

The more sobering independent evidence is the earlier JevBench v1.4 sealed report: Open-Jev 9B’s sealed accuracy was 29.9%, far below its 77.5% public-slice result. JevBench v1.5.4 later changed the protocol and reports chance-corrected competence rather than the same raw accuracy, so the two versions should not be read as a trend.

The failures remain in the repository

Open-Jev’s strongest feature may be its refusal to erase bad results:

  • A natural-support pilot put the released 2B behind BM25: 166/256 versus 211/256.
  • The support pilot retained 14 errors among 88 accepted rows, so human review stays on by default.
  • A v7 synthetic adaptation improved selected slices but failed every predeclared safety gate.
  • That v7 candidate still selected dangerous actions and accepted all 12 old outside-window Choice cases.
  • Later input auditing found legacy training rows that matched observed development fingerprints, limiting claims about those slices.
  • Prefix caching changed probabilities beyond tolerance on several workloads even when selected decisions matched, so it remains off by default.

A correct typed response is only a valid data structure. It is not evidence that the selected action is safe or correct.

Runtime guardrails in the implementation

The source includes practical safeguards that are easy to miss in benchmark tables:

  • Inputs exceeding the configured token limit raise an error instead of silently truncating.
  • The loader refuses to serve when sharding leaves required parameters as unloaded meta tensors.
  • Quantized and alternate-dtype paths are labeled numerically incomparable to released results.
  • Training identity binds arguments, source revision, data hash, selected row IDs, optimizer settings, and implementation hashes.
  • Resume checkpoints are checksummed and rejected when identities or cursors differ.
  • Held-out splits, calibration, and training are represented separately.

Local inspection result

I downloaded the public source snapshot, compiled jev/ and scripts/, and ran the suite from the repository root with PYTHONPATH=..

  • 1,548 tests ran.
  • 1,456 passed.
  • 90 skipped.
  • One race-oriented restoration-audit test expected a file-change error that was not raised.
  • One training-initialization test errored because the ZIP snapshot has no .git directory and the implementation requires a source checkout commit.

The first issue may be timing- or filesystem-sensitive; I did not diagnose it further. The second is a consequence of testing a GitHub ZIP rather than a Git clone. Either way, the suite did not return success, so this capture does not claim an independently passing checkout.

Takeaway

Open-Jev 27B is substantial because it opens the whole stack: scorer, training recipe, adapters, decision head, calibration, deployment code, datasets, raw-ish audit artifacts, and known failures. Its public-slice result is close to Jev’s recorded result, but the project’s own evidence argues against treating that as solved decision intelligence.

The honest status is more useful: a capable open decision model with reproducible artifacts, incomplete independent 27B evaluation, workload-specific weaknesses, and safety boundaries that still need ordinary software controls.