PROBE tests whether coding agents find defects that maintainers later repair. Its reference set contains 2,457 bugs fixed over eighteen months in twenty repositories, all already present in the starting snapshots.
The authors split audits into smaller file groups, provide repository context and match reports against historical fixes.
- The best individual full-set configuration recovers 20.8% of reference bugs. At most 8.1% of reported findings match the historical list.
- Those non-matches are often real: a separate model panel confirms 84% of sampled unmatched findings, excluding inconclusive judgments.
- A blinded human check agrees with the panel on 93% of 131 conclusive comparisons.
- Agents favor local arithmetic and comparison errors. Maintainers more often repair missing support for legitimate inputs and public interfaces.
Caveat: Historical matching is not a false-report rate. The reference set is incomplete, and a separate human audit rejects twelve of one hundred sampled entries.
The paper’s five-to-ten-year bug-pool estimates depend on extrapolation. Its appendix says the observed data provide no finite upper bound on total bugs. These are author-reported results, with dataset download access still unverified at capture time.