jev-zh-tw-eval is an unusually candid evaluation of crh225/plumb-4b, a local Jev-compatible decision model served through Ollama’s /v1/systemone endpoint. The repository combines Traditional Chinese classification, option-count stress tests, prompt injection, paraphrase sensitivity, confidence checks, raw aggregates, and a standard-library CLI.

I inspected the README, implementation, fixtures, recorded results, and validation reports. I also ran the offline release verifier and all 80 CLI tests successfully. I did not rerun model inference, so the accuracy and robustness figures below are the author’s recorded measurements—not independently reproduced model results.

Useful performance, but strongly task-dependent

  • On 231 public JevBench questions, the project records 203/231 (87.88%) using its local CLI scorer; this is explicitly not the official JevBench Score.
  • On the same 100 MASSIVE zh-TW intents with Traditional Chinese option descriptions, recorded accuracy fell from 96/100 with 2 options to 93/100 with 8 and 81/100 with 26.
  • A two-stage strategy for the full 60-class intent task reached 74/100, only modestly above a direct top-one strategy at 70/100.
  • Historical Taiwan-news publisher classification reached 138/192 (71.88%) for CNA and 211/320 (65.94%) for LTN using title plus summary.

Those results are not one universal “model accuracy.” They show that the question, candidate set, labels, language, and task distribution materially define the result.

High probability still includes wrong decisions

The repository’s 24-case synthetic weakness probe recorded 18 correct answers. More importantly, 3 of the 12 answers whose selected-answer probability was at least 0.9 were wrong.

That distinction matters operationally:

  • typed output means the response matched the requested structure;
  • a high selected-answer probability means the returned distribution was concentrated;
  • neither fact establishes that the decision was correct;
  • the API’s separate confidence field is not interchangeable with the maximum answer probability.

A production threshold therefore needs workflow-specific labels and an independent holdout. It should not be copied from a small synthetic suite.

Prompt injection reaches the decision layer

In the first 18-case synthetic injection suite, a disguised note pushed Plumb-4B to the attacker’s target answer in 7/18 cases (38.9%). Two target answers carried selected-answer probability of at least 0.9.

A second suite recorded:

  • clean: 18/18 correct;
  • attacked: 13/18 correct, with the attacker’s target reached in 4/18;
  • “guard instruction + JSON wrapper”: 9/18 correct, with the target reached in 8/18.

The project correctly refuses to attribute the last change to the guard instruction alone because the state format changed at the same time. It also distinguishes “the answer became wrong” from “the attacker achieved its chosen target.”

The practical conclusion is narrower and stronger than a generic security slogan: untrusted text inside the state can influence a decision model, and a protective sentence does not turn model output into an authorization boundary.

Wording and batching are part of the model input

Small representational choices also moved outcomes:

  • Four paraphrases of each of 27 synthetic business cases produced a mean within-case probability range of 0.201 and a maximum of 0.805.
  • 5 of 27 cases crossed the 0.5 boundary under paraphrasing.
  • At batch size 64, choice agreement with single-question execution was 26/30 in the recorded batch experiment.
  • Negated questions did not behave like simple probability complements.

A validated workflow must freeze—or deliberately test—question wording, option descriptions, option count, candidate order, and batch composition. “Semantically similar” is not an adequate regression criterion when a small wording change can switch the downstream branch.

The local harness is real and inspectable

The repository is more than a result table:

  • The CLI supports Noul, Choice, and Score requests against local Ollama or the paid TypeSafe API.
  • Local requests enforce 2–26 options, a 64-question limit, request-size checks, endpoint health caching, and a pinned model digest.
  • The evaluation scripts preserve raw answers and aggregate them separately.
  • verify_release.py checks UTF-8/LF files, Python syntax, JSON, relative links, 2,587 valid fixtures, 200 expected 27-option rejections, and selected aggregate anchors.
  • In my local inspection, the release verifier passed and the full 80-test CLI suite passed in 37.8 seconds.

That verifies the packaged code and committed artifacts. It does not authenticate historical API responses or reproduce the model runs.

Limits the project keeps visible

  • Synthetic cases use one author’s labels and do not estimate a deployment population.
  • Several sentiment and safety datasets are Simplified Chinese, not Traditional Chinese.
  • Public benchmark questions may have appeared in training.
  • The author’s 131-item test split was also used for calibration, so it is not a contamination-free final holdout.
  • News categories overlap, historical RSS text is not fully redistributable, and the publisher labels are not a newly adjudicated semantic benchmark.
  • Shared GPU load makes the recorded latency unsuitable for clean hardware comparisons.
  • No security audit was performed here.

Takeaway

This is evidence for using a local decision model as a measured assistant or router, not as the only authority for refunds, deletion, external messages, or tool calls with side effects. For those actions, the repository recommends deterministic rules or human confirmation.

The strongest result is not any headline accuracy number. It is the accumulated evidence that typed probabilities remain sensitive to framing, candidate count, hostile text, and wording—and that high probability can still be confidently wrong.