A coding assistant’s failed test can expose a bad task as readily as bad code. Soham Dan and colleagues at Scale AI and Reflection AI describe repairing SWE-Bench Pro and closing ways to pass without solving the assigned problem.
Their process combines automated audits, expert adjudication and blind re-implementation. A second engineer must solve a revised instruction without seeing the reference patch before the repair ships.
- Twenty-three contracted engineers spent 404 hours on review. Of 731 original tasks, 391 were repaired and 89 removed, leaving 642 tasks and a 51-task Hard subset.
- During model work, the software permits network access only to the model endpoint. This prevents fetching upstream fixes from code hosts.
- Grading applies the submitted patch to a fresh container. Replacing a local executable or forging a dependency outside the patch can no longer earn credit.
- The best reported system resolves 99.4% of the full set. Hard scores range from 52.9% to 98.0%, but that subset still cannot reliably rank the top four systems.
The released dataset dates V2 availability to September 22; the paper was submitted October 16. This is an explanation of the repair and evaluation protocol, rather than a first release of the tasks.
There is an important limit inside the paper. Its separate comparison of three agent wrappers uses grading in the agent’s own container, unlike the clean-container main results. The introduction’s suggestion that wrappers change cost but not accuracy also needs qualification: Inkling’s scores vary by 7.5 percentage points.
Caveat: These author-reported results approach the test set’s ceiling, and blocking downloads during evaluation cannot rule out memorized fixes from model training.