Kaiser Sun and colleagues compare reasoning-effort settings across eight models and four task sets. Higher effort often improves aggregate accuracy, while spending more on tasks that cheaper runs already solve.
The study measures whole executions, including repeated input context. Its terminal tests contain 89 tasks; its repository-repair pilot contains 44 instances. Each agent task runs once per effort setting.
- On terminal tasks solved at every tested effort, GPT-5.6-sol costs 2.51 times more at extra-high than low effort. Claude Sonnet 5 costs 3.04 times more.
- Original repair containers expose reference fixes through repository history. Removing that history and restricting network access lowers GPT-5.6-sol’s extra-high success from 77.3% to 59.1%.
- Starting low and retrying at the highest tested effort after verified failure saves a median 15.5% across models on corrected repairs. Seven of eight models cost less, with both attempts charged.
- Repeated-run controls show a trade-off: low-to-high costs less than high-to-high, but high-to-high remains more successful.
The retry policy is reconstructed from recorded runs. It uses an external verdict and a fresh second attempt. Asking the model to choose effort before solving offers no consistent advantage.
Caveat: Selective escalation requires a reliable external correctness check, and observed reversals can reflect sampling variability.
The code separates usage, prices and modeled waiting costs. Waiting-cost estimates assume a blocked user; they are not measured lost wages. Several authors work at Together AI, which serves the tested downloadable models.