Trail of Bits reviewed its merged security patches and confirmed 21 regressions across more than 130 pull requests. Its new agent workflows turn that experience into explicit checks for whether a fix works and whether it breaks previously supported behavior.
Anish Naik and Marcelo Morales describe two plugins: patching-bugs develops and validates repairs; regression-hunt investigates defects introduced by changes. A regression is a change that breaks behavior that previously worked and was supported.
What the review found
The 21 confirmed regressions comprised 16 functional defects, one performance defect and four build or test defects. The authors say none broke a security protection.
In one review run, those confirmed regressions came from 58 candidates, illustrating how much human review false alarms can consume. These are results from Trail of Bits’ own patching work, not a measured failure rate for AI-generated patches generally.
Evidence status: an internal patch review and preliminary tool evaluation, not a report of attacks. The relevant exposure is a maintainer accepting a repair that changes reachable, supported behavior elsewhere. The article supplies no product-wide affected-version list or new CVE requiring an emergency update.
Make the fix earn its evidence
The patching-bugs workflow requires the same reproduction to show three results:
- The original code fails. Establish the reported bug through a real interface.
- The patched code passes. Show that the repair removes the failure.
- A complete revert fails again. Removing the repair should bring the bug back, helping separate the patch’s effect from unrelated test or environment changes.
This checks the repair’s causal role. It does not establish that every supported behavior still works.
That broader question belongs to regression-hunt. It traces changes to reachable interfaces, reproduces suspected regressions, assigns an independent replay to a fresh agent, and judges the retained evidence. A plausible explanation without the required running reproduction stays inconclusive.
Fewer false alarms have a cost
On its set of 21 confirmed regressions, Trail of Bits reports 25% fewer false-positive findings than a baseline Codex agent asked to review for regressions. It gives a 95% confidence interval of 14% to 38%.
The stricter evidence rules also left some known regressions classified as borderline. Fewer false alarms therefore came with a reduced ability to confidently report some real defects. The authors say a separate evaluation of patching-bugs is still being built.
What maintainers can use now
Write down supported configurations, trust boundaries and the evidence needed to accept a bug report in a THREAT_MODEL.md. Put build steps, patch scope and regression-test expectations in PATCHING_GUIDELINES.md or AGENTS.md.
Require raw before-and-after evidence tied to exact revisions, plus an independent replay, before accepting an agent’s conclusion. Keep execution in an authorized, isolated test environment with explicit resource limits. The plugin README says missing execution or independent-agent capabilities leave findings unresolved.
Trail of Bits develops these tools through Patch the Planet, its joint initiative with OpenAI. The company is evaluating workflows it built for its own engineers.
Caveat: the reported improvement comes from the developer’s small, internally assembled evaluation set; it does not establish general regression coverage.