BuggyMLE deliberately damages working machine-learning repositories to test whether coding assistants repair every fault. John Wu and colleagues separate restoring an application’s score from fixing its code.
Their fixed task set covers 22 competitions at four difficulty levels. Full repair requires at least 99% metric recovery, execution through the affected modules, and a genuine fix at every verified bug site. A model grader checks those sites against recorded defects.
- Claude Opus 5.5 fully repairs 95.5% of single-bug tasks but 9.1% of the hardest tier. That tier expands repository size and inserts 23–60 faults despite its “50 Bugs” label.
- On ten-bug tasks, Opus restores the target score on roughly 64% of repositories but fully repairs only 13.6%. Improvements elsewhere can compensate for remaining faults.
- Training Qwen3.5-9B on an easy-to-hard repair curriculum raises success on 300 SWE-bench Lite issues from 24.3% to 33.0%.
- Generating fresh machine-learning solutions shows mixed transfer. Better repair behavior does not establish generally better research automation.
Caveat: These are author-run tests with injected faults and partly model-graded repairs. The reported solve rates are specific to the task set, tools and per-task budgets.
The practical check is to verify the affected path and each defect alongside the final score. Faults that leave local validation scores unchanged are especially easy to miss.