Testronaut tested Jev as both a guardrail and a model selector for browser-testing agents. The useful result is not “always choose a cheaper model”: its reported cross-provider router produced longer token trajectories than the strongest fixed executor.

I read the complete report and checked its public condition CSVs. I did not rerun the missions or make live Jev calls; all performance figures below are Testronaut’s reported measurements, not independently reproduced results.

Separate the check from the switch

  • The first experiment used 70 aggregate runs, covering 210 mission-file attempts, with five randomized blocks per condition.
  • Guarded GPT-5.5 passed 15/15 attempts, versus 5/15 for unguarded Terra, while using 13.7% fewer executor-model tokens.
  • The reported paired interval for that token change was −19.8% to −7.5%; mean duration nevertheless rose from 293 to 416 seconds.
  • That headline comparison changes both the executor and the guardrail. The same-model guarded Terra comparison improved passes to 8/15, but its token-change interval included no improvement.

This is suggestive evidence for completion/evidence checks, not clean proof that Jev alone caused the headline gain. Typed output is not a correct decision, and a completion judgment still needs evidence.

The cheap selector picked costly trajectories

In the second experiment, Testronaut isolated 120 mission executions. Its fixed GPT-5.5 condition and GPT-5.5 ↔ Gemini Flash-Lite router both passed 15/15, but reported mean executor-model tokens rose from 87,422 to 104,897 under routing.

The selector itself averaged 621–724 tokens across routing conditions. Small selector overhead did not rescue the longer executions; Jev chose a Gemini candidate in 50/60 routed missions despite GPT-5.5 being the strongest fixed control on this workload.

Count Jev too—and do not call tokens dollars

The first experiment’s guarded GPT-5.5 condition also reports 174,910 mean Jev tokens separately from its 258,635 executor tokens. The advertised reduction is therefore not a reduction in all raw inference-token volume. Different token prices could still make the workflow cheaper, but the report does not calculate price-weighted spend.

Other limits matter: small repeated mission sets, private raw browser reports, and six initial Gemini HTTP 500 responses rerun before the final completed dataset. Testronaut recommends conservative routing and history-informed capability estimates—not zero-shot cross-provider selection as a default.

The transferable lesson: compare successful-task cost, completion evidence and wall-clock time, not merely the price of the next call.