Towards Data Science compares seven open Jev-style projects with a small language model generating JSON. The article supplies a companion notebook by Lambert Leong, pinned implementations and saved results. It evaluates open alternatives; hosted TypeSafe Jev is absent from the comparison.

Keep the comparison small and specific

The author reports two BANKING77 customer-support tests, each repeated twice:

  • Fixed routes: 200 requests share eight categories. Kev reaches 98% accuracy, Laya 92%, Verdict 86.5%, and Qwen3-0.6B generating JSON 83%.
  • Changing menus: 100 requests supply four shuffled choices. Kev reaches 98%, Verdict 89%, Laya 74%, and JSON generation 66%.
  • The three alternative ways of reading the same Qwen checkpoint’s scores trail JSON generation on both tests.
  • NanoJev’s game-trained checkpoint reaches 17% on the changing-menu banking task, despite accepting its request format.

These are author-reported inference results, not independently rerun measurements. The notebook’s embedded accuracy summaries match the article.

The changing-menu test also changes the target categories. It therefore leaves the effect of option order entangled with task difficulty.

Separate formatting from choosing correctly

The author reports nine changing-menu responses rejected because Qwen wrapped valid JSON in Markdown fences. Local inspection confirmed nine such failures in the notebook’s saved error examples.

  • A strict parser rejects those wrappers; stripping them would create a different configuration to evaluate.
  • Typed output removes that formatting failure while leaving wrong in-menu choices possible.
  • AnyJev’s tested path rotates candidate positions across multiple evaluations. Zero generated answer tokens alone says little about how much computation it performs.

Check the labels before trusting the ranking

The saved errors include “Where in the app can I find my PIN?” with the target get_physical_card. That target’s transmitted description is simply “get physical card.”

A read-only check found the same text and target in the exact pinned dataset release. Its file hash matches the benchmark’s expected identity. The example raises a source-label problem, rather than showing that this benchmark changed that record.

A mislabeled reference can reward an implausible answer or penalize a sensible one. This spot check cannot establish how widespread the problem is; the category mapping needs an audit before the ranking supports deployment.

Treat timings as configuration measurements

The author reports warm median times of about 125–130 ms for Kev, 17–18 ms for Verdict and 331–357 ms for JSON generation. Separate GPU sessions, different numerical precision and different timing boundaries limit the comparison.

The dollar estimates assume $2.50 per GPU-hour, excluding loading, idle capacity and production infrastructure. They are modeled compute costs, not observed API bills.

Verification: read the full article and notebook; checked embedded summaries, format-error examples, parser and dataset code, plus one hash-verified source record. Models, accuracy and latency were not rerun; confidence thresholds remain unvalidated.