Vinicius Lana’s jev-eval-agent tests who should choose an assistant’s next tool. One mode exposes the full catalog to a language model. The other asks Jev to select a tool, then lets the language model supply its arguments.

The repository includes the implementation and saved traces. Recomputing those records reveals a tradeoff: smaller language-model inputs accompany more model steps and weaker coverage of required actions.

Follow the saved results

An offline count of the author’s artifacts finds 84 task records: six Portuguese-language tasks, seven model configurations and two modes. The catalog contains 100 mocked tools with deliberately similar alternatives.

Across those saved records:

  • Full required-tool coverage: 41 of 42 direct-mode tasks, versus 30 of 42 Jev-routed tasks.
  • Language-model calls: 157 in direct mode, versus 208 with Jev routing.
  • Tool calls: 202 in direct mode, versus 165 with Jev routing.
  • Estimated language-model cost: about $8.07 versus $1.52, using the repository’s pricing calculation. Jev’s own bill is excluded.

These counts reproduce the saved artifacts, not live model behavior. The Jev mode also lowers or disables the language model’s reasoning effort and changes its instructions, so the comparison changes several variables together.

Inspect the stop decision

The router asks Jev two questions: which tool comes next, and whether every requested action is finished. If Jev selects respond_to_user while its completion probability is below the configured cutoff, code exposes the next-ranked tool.

That gate needs enforcement beyond tool visibility. In the saved Gemini meeting task, the router eventually exposes a contact lookup, yet the language model gives its final summary without performing it.

Earlier in the same trace, the model invents attendee email addresses. The mock meeting tool accepts them. Selecting a typed tool name leaves both argument correctness and completed work to verify.

Grade the resulting work

The evaluator counts completed calls from required tool families. It also tracks look-alike wrong tools and extra calls. It leaves argument correctness, action ordering and real external effects largely ungraded.

A run can therefore finish with a successful execution status while missing a requested step. Fewer tool calls can mean less work was done.

For a stronger follow-up, keep reasoning effort fixed, repeat each task, charge both models and check the final simulated state. Those are editorial recommendations, not experiments run by this capture.

Verification: inspected the README, Jev adapter, tool exposure, mode instructions, mocks and evaluator; recomputed totals from saved records at commit 037de1120c84b4b63cdf748e2acf258ff66d7731. No dependencies, agent runs or live API calls were executed.