Sergio Villani at Google Cloud AI tests Jev as a router for short coding-agent questions. His experiment compares model settings as well as routing policies, which exposes a trap: a supposedly cheap path can cost more.

Ask about the work the answer requires

A broad simple versus complex question sent nearly everything down the simple path. Villani replaced it with one Noul question, a yes/no probability: does answering require several sequential calculations, simulation steps or deductions?

  • A frozen keyword-and-length rule matched the hand-labelled difficulty on 70.6% of 34 held-out tasks, Google reports.
  • Jev matched all 34 held-out labels, reaching 48 of 50 across development and held-out questions.
  • Longer extraction requests fooled the length rule; short code traces escaped its keyword list.

These are difficulty-label results. A typed label alone cannot establish that the selected model will answer correctly.

Lower thinking directly before adding a router

Google reports that limiting Flash to 300 output tokens barely changed its thinking-token use on easy questions. The cap broke 7 of 25 hard tasks instead.

Routing easy tasks to capped Flash recovered correct answers across the suite, but cost slightly more and ran slower than uncapped Flash alone. The model still did the expensive thinking.

The stronger baseline was Gemini 3.8 Flash with low thinking, which Google says solved all 50 tasks. A Jev policy using Flash-Lite for simple work and low-thinking Flash for harder work also solved all 50, at lower reported cost and response time. The keyword router solved 48.

Short tasks and long agent sessions have different economics

  • The benchmark uses short, standalone prompts with simple automatic grading.
  • Long sessions repeatedly send repository context and tool definitions; switching models can require separate caches or lose a cache discount.
  • A router’s bill includes classification, downstream model calls, retries and failed work.
  • Compare against the cheapest single-model setting that already meets the quality target.

Caveat: these are Google’s single-run results on a small suite, not independently reproduced performance. The full benchmark code was still awaiting open-source review when captured.