Musa Cim and colleagues saved intermediate coding-agent patches and checked them against benchmark tests after each run. Their measure, termination lag, counts the steps between a first passing patch and the end of the run.
- In successful runs in their main settings, a median 35–60% of steps followed the first patch that passed the hidden tests. Most later steps were verification or exploration.
- Among runs that reached a passing patch, 1–8% later broke it and ended with a failing patch. In other runs the agents repaired a regression before stopping.
- A second framework showed a similar post-fix share for Qwen on SWE-bench Verified: 37% versus 35% in the main framework.
The agents could not see the hidden tests. An ideal stop at the first passing patch is an oracle, not an available stop rule. Test passage also does not prove a patch meets every real requirement. Each task was run once, and step counts do not directly measure cost.