Vladimir Beskorovainyi’s Ru-Jev scores caller-supplied options using a local encoder. Its model card describes an independently trained, Russian-first alternative in the Jev category, with open weights and inspectable training and evaluation code.

The release’s strongest evidence concerns a practical failure: changing a category’s wording can change both its accuracy and its apparent certainty.

Rename the options, keep the questions

The author compares earlier and current releases on the same routing questions. Local recomputation from his saved responses matched these five-candidate results:

  • With ordinary Russian category words, accuracy moves from 95.6% to 96.0%. Both releases trained on that wording style.
  • With anglicized jargon, accuracy moves from 63.3% to 85.5% across the full label set.
  • On that jargon test, wrong answers above 0.8 confidence fall from 85 to 44 per 1,000 questions.

The full jargon set contains wording overlaps with training. The card separately reports stricter slices and tests with more candidates. Treat the full-set figures as one defined test, rather than a blanket robustness score.

Better choices can still cross a bad threshold

The author reports that confident-error counts rise on some tests offering more choices, despite improved accuracy. Abbreviations and opaque codes remain weak points.

  • Describe categories with meaningful words and test the exact vocabulary your application sends.
  • Refit temperature, the scaling value that adjusts confidence, on held-out examples from that workload.
  • Include an explicit “none of these” option when the supplied choices may omit the right answer.

Typed output fixes the answer’s structure. It leaves the correctness of the choice and its probability to be tested.

Preserve the evaluation split

The project documents earlier comparisons invalidated by shifted training/validation splits and changed option wording. Rebuilding the data can introduce training examples into validation.

On the author’s preserved, directly comparable task rows, local checks found 8,120 correct answers out of 10,157 for each release. Each wins on 840 rows the other misses. The matched aggregate supports equal accuracy there, alongside different individual decisions.

The author also warns that this is a comparison between trained releases, with several changes between them. It does not isolate label augmentation as the cause.

Load the scoring head explicitly

The card’s loader combines the encoder with a separately saved scoring head. It warns that the stock zero-shot pipeline initializes a fresh head and returns misleading numbers.

The model card reports CPU timing on an Apple M4 and claims no speed advantage over its Laya comparison. Those timings were not reproduced.

Verification: inspected the model card, training, scoring, serving and evaluation code; recomputed selected results from the author’s saved scores. No model inference, retraining, latency measurement or live Jev calls were performed.