Torana tested Laya as a local model selector for coding agents. Its proxy asks whether each new user turn belongs on a cheaper or stronger model. The experiment used Laya’s English checkpoint, an open Jev-style classifier; hosted Jev was never tested.

A working connection can still make poor decisions

Torana reports these results from one Mac, using its own synthetic prompt labels:

  • 112 ms median per warm classification across 25 sequential requests.
  • The default question agreed with the author’s difficulty rubric on 21 of 48 prompts.
  • Across five question variants, no answer cleared the plugin’s 0.8 confidence threshold.
  • Renaming options and changing their order changed 13 of 48 decisions, despite unchanged descriptions.

The successful Codex, Pi, Oh My Pi, and OpenCode runs checked compatibility. They did not establish better coding outcomes or lower bills.

Know which confidence field you threshold

Laya supplies both answer_confidence, the chosen option’s probability, and confidence, a measure of how concentrated the answer distribution is. The plugin checked the latter.

Torana’s best chosen-option probability on the default question was 0.64, paired with 0.26 confidence. Lowering the gate would activate more weak decisions without improving their agreement with the rubric.

The classifier also lacked the current model, code, tool output, and conversation history. An option saying “the current model remains suitable” asks for evidence the request never supplies.

Reject incomplete judgments and measure actual outcomes

  • Torana reports that a synthetic 0.99-confidence response marked truncated still produced a suggestion. Truncation and abstention need explicit handling.
  • Local tool errors can trigger escalation even when the real problem is missing Docker or a broken network.
  • Switch only at a defined boundary, and include the cost of rebuilding the model’s prompt cache.
  • Compare completed tasks against always-cheap and always-strong baselines before enabling automatic switching.

In Torana’s separate, tiny comparison, Flash answered 10 of 10 questions correctly and Pro 8 of 10. Both had thinking disabled. That author-run result challenges the assumed benefit of upgrading; it cannot rank the models generally.

Torana kept the router in shadow mode: recording suggestions while leaving the selected model unchanged.

Caveat: author-reported experiments, not independently reproduced here; no savings were measured. Typed output is not a correct decision.