E. Kelly Buchanan and colleagues treat coding benchmarks as maintained software. Instructions, containers, grading tests and time limits can all change what a score measures. Their paper examines the existing Terminal-Bench 2.1 update.
The workflow combines task files, execution records and community reports. Models propose diagnoses and repairs; known-correct reference solutions and human maintainer review check them.
- The paper counts 28 revised tasks among 89. The current repository introduction instead says 26, a discrepancy to resolve before citing the release’s exact scope.
- Across 19 agent configurations, scores move by up to 12 percentage points and positions by three places. Broad ranking agreement stays high, with a correlation of 0.958.
- Extending time limits up to 32 times closes a roughly 20-point gap between two harnesses running GPT 5.4. A harness is the software running a model’s tools, files and commands.
- For GPT 5.4 mini, a roughly 30-point gap shrinks to five points. Giving slower systems longer changes the share of tasks solved, rather than showing faster answers at the original budget.
The audit’s 57.1% precision and 71.4% recall measure agreement with repairs it helped produce. They are retrospective checks, not independent proof of automated defect discovery. A separate Harbor-Index sample confirms 12 of 14 flags but also reveals missed defects.
Caveat: The task-count discrepancy and resource changes need checking against the exact artifact. The failure analysis also filters out infrastructure errors affecting roughly 27% of upstream trials.
For a reproducible comparison, pin the benchmark revision, container, harness settings and resource budget alongside the model. The reported score shifts come from changes to the measurement setup.