Silas Liu’s unusually careful comparison starts with architecture, predicts behavior, and then tests those predictions. The result is more useful than another leaderboard: it explains why three open System One models fail differently and uses those signatures to infer how closed-source Jev probably works.

Three open designs

  • Laya puts options before the state in one ModernBERT sequence and scores each option’s [MASK] embedding.
  • Lev mode A turns the task into multiple choice: state first, options last, then reads the next-token probabilities of the option letters. A LoRA adapter sharpens that behavior.
  • CLM embeds the state and every option separately, transforms them with two trained MLPs, and compares them by cosine similarity.
  • These choices determine whether options can be cached, whether list order matters, and whether a one-word edit survives into the decision.

Failure modes follow the architecture

  • At 77 options, Liu reports that shuffling changed Laya’s answer for 74–79% of tested states, versus 5–6% for Lev mode A and 0% for separately encoded CLM.
  • Laya’s default token budget can silently truncate the input. With 150 options, 99 of 100 questions were rejected; the remaining request kept the options but removed all 32 state tokens.
  • On controlled minimal edits involving count, negation, roles, time, and quantifiers, CLM usually failed to switch because its state is compressed before it encounters an option.
  • Misleading option names dominated Laya even when descriptions were correct. Lev mostly followed the descriptions; CLM ignores names whenever descriptions exist.

The Jev inference

  • Jev’s probabilities changed more under option reordering than under repeated identical calls, arguing against independently encoded options.
  • Option position and distance had no measurable effect, unlike Laya’s options-before-state layout.
  • TypeSafe’s reported token counts imply a fixed prompt of roughly 270 tokens, options serialized with per-option formatting, and a state charged once across several questions.
  • Together, those observations fit a Lev-like decoder: read the state, read coded options, take probabilities at the answer position, and branch several questions from one shared state representation.
  • That is an inference from behavior and billing metadata—not disclosure of Jev’s weights, prompt, or serving code. A different implementation could produce the same observations.

What the measurements do—and do not—show

  • On Liu’s 100-state BANKING77 subsets, Jev accuracy declined from 1.00 with two options to 0.77 with 255; Lev remained ahead at large option counts, but its model card says BANKING77 was in training.
  • Jev handled nearly every hand-written minimal edit and control pair, but these are small, authored tests rather than a broad independent benchmark.
  • The reported Jev latency stayed near 250 ms as the option count grew. Client latency includes network and queueing, so this does not isolate model compute.
  • Liu publishes four notebooks with pinned commits and matched seeds. I reviewed the full writeup and its method but did not rerun the GPU experiments or paid Jev calls.

The practical lesson

  • Benchmark scores are not portable guarantees. Test the condition your architecture is likely to mishandle: long lists, reordered options, contradictory names and descriptions, minimal edits, or out-of-domain labels.
  • Typed probabilities still come from a model whose answer can flip with presentation. Schema validity is not decision validity.
  • For large stable option sets, cached independent embeddings buy predictable scaling but may lose fine relational detail.
  • For decisions that hinge on a word, role, or comparison, reading state and options together appears materially stronger in these tests.

“Knowing why each design behaves as it does is what allows such variants to be built deliberately, rather than discovered by trial and error.”

This is the strongest Jev analysis so far because its claims are falsifiable: architecture produces a predicted signature, experiments look for that signature, and the closed model is described with explicit uncertainty rather than launch mythology.