Yuchen Ling and colleagues test what happens after a cheap decision becomes an action. Their Jev study follows household and Android agents through complete tasks, counting failed attempts and supporting language-model calls—not just the speed of selecting the next step.

I read the complete preprint, including its methods and appendices. All experimental figures below are the authors’ results, not independently reproduced here.

The decision gets cheaper; the workflow may not

The authors report 4,500 executions across five agent families, comparing jev-1.13.0 with workflows using gpt-5.6-luna for decisions and auxiliary generation.

  • Decision-level estimated cost falls 66.1–78.4%, and decision time falls 73.9–81.5%, in their integrations.
  • Household-task success remains close to baseline; the shared-candidate control matters because offering a menu changes the task itself.
  • Android T3A success falls from 57.47% to 44.25%; FastAgent falls from 50.00% to 31.90%.
  • FastAgent’s total task time barely changes: more interactions and supporting work absorb the faster decisions.

The authors’ task-cluster uncertainty intervals support the T3A and FastAgent success losses. These are results for particular integrations, not an isolated ranking of model intelligence.

Charge failed attempts to the successes they deliver

In the authors’ T3A results, estimated model spending per attempt drops 17.1%, but spending per delivered success rises 7.7%. The interval around that cost increase includes zero; the point estimate is a warning, not a demonstrated universal penalty.

  • Count all attempts’ expenditure, then divide by completed tasks—not just the cost of runs that happened to succeed.
  • Include auxiliary reasoning, reflection, and environment interaction in timing.
  • Keep model-token estimates separate from infrastructure, human review, and total operating cost.

The authors report that Reflexion-J’s unsuccessful executions consume 53.3% of its inference expenditure, despite representing only 2.0% of executions. A small failure tail can dominate the bill.

Recovery is a capability, not an optional extra

The paper’s traces show agents persisting with a handtowel when the task requires a cloth, or declaring Wi-Fi enabled after clicking an ambiguously labeled switch. More reflection can reinforce the wrong premise instead of correcting it.

  • Expose target identity and action prerequisites, not just clickable options.
  • Detect repeated lack of progress and change strategy.
  • Verify the requested final state independently of the agent’s DONE decision.

Typed output is not a correct decision—and benchmark acceptance is not always real-world completion. One map task passes a name-based evaluator without establishing geographic correctness.

The study uses text-only observations, fixed task instances, repeated runs with a shared seed, and estimated token prices. FastAgent also changes from potentially batched tool calls to one tool per step. Those boundaries limit causal attribution and transfer to newer multimodal Jev versions.