Huawei researchers examine about 6,000 coding-assistant runs across eight downloadable models and two surrounding software setups. They reconstruct intermediate patches, grade them with hidden tests and calculate spending at each step. The question is whether work after a passing self-test earns its cost.
Their simplest practical rule stops after two passing tests on an unchanged patch. Hidden correctness tests score the result without giving their answers to the assistant.
- In a replication of 386 new Qwen3.6-27B runs, the rule saved 22.8% in calculated inference cost. The fixed-task success rate fell one percentage point.
- That average loss sits inside a two-point tolerance, but its uncertainty does not. The prespecified test failed to establish that quality stayed within the two-point margin.
- Training the assistant to imitate the rule saved 30.0% across six training seeds, with a 1.2-point average quality loss. Quality preservation remained uncertain.
- The software around the model matters. The same stopping rule saved 54.2% in SWE-agent but lost 2.8 points, where later edits more often repaired an incomplete fix.
A scoring audit materially changes the interpretation. Earlier working-tree comparisons omitted staged edits. Correcting some of those cases removes an apparent quality-assurance pass and withdraws earlier claims that selecting the last passing patch improves Qwen results.
The evaluation method is useful independently of the rule. Comparing stopping with completion on the same recorded run reduces noise. It works only when the intervention preserves preceding behavior and the record faithfully captures the relevant patch states. A new prompt or fine-tuned model still needs new runs.
Caveat: Passing the assistant’s own tests does not establish independent correctness. The patch correction remains incomplete, and matching final reconstructed patches cannot verify every intermediate state. Costs price local token records and exclude sandbox computing and elapsed time. This is evidence for evaluating stopping policies, not a universal instruction to stop after two green checks.