Zichuan Xu and colleagues hold an agent’s task fixed while changing its tools, workflow or returned records. Their ToolBottleneckBench study tests 21 model-and-provider configurations. It separates the model’s choices from the limits imposed by the software running its tools.
- Growing the visible menu from eight to 128 tools cuts success by 12.3 percentage points under the main setup. That setup allows just 1,024 generated tokens per response, sometimes including hidden reasoning.
- In six configurations that frequently hit this ceiling, raising it to 16,384 tokens shrinks the menu penalty from 43.7 to 2.7 points. These figures come from a separate repeated experiment.
- Among 3,725 failed episodes, 91.5% end at an execution limit, including early stops when success becomes impossible within the remaining calls. Every failure labeled “tool selection” happens before the model executes any call.
- Better tool retrieval and handing predictable call chains to ordinary code both help. Their gains come from making the right tools available or doing work outside the model.
A failure label can describe where a check stopped while pointing to the wrong repair. Record the terminating limit alongside the label, then compare budget changes, routing and model switches on the same tasks.
Caveat: these are author-reported results on synthetic tasks under tight limits. In the external ToolSandbox check, larger menus clearly cost more tokens, but the measured success loss remains uncertain.