DistDebug-Bench separates finding a failure’s cause from generating a patch. Coding assistants receive an incident report, available evidence and a repository snapshot from before the fix. They must identify the causal code blocks.
- The authors curate 120 real bugs from eleven distributed systems in five programming languages, with 218 human-annotated causal blocks. Almost half the incidents require more than one block.
- GPT-6 Astra with Mini-SWE-Agent recovers 83.9% of annotated files but 67.7% of causal blocks. Its block precision is 42.7%, meaning many predicted blocks fall outside the annotated causes.
- Each configuration averages the first three schema-valid scored attempts for each incident. Missing or invalid attempts can be replaced; these are not unconditional success rates for every attempted run.
- Trace inspection finds early commitment to plausible local explanations and search commands that expose different evidence. Proposed improvements to context and hypothesis management remain design recommendations, not demonstrated fixes.
The released benchmark, traces and leaderboard support checking the authors’ results. Appendix G also finds scoring false negatives: some labels use coordinates from a different revision, and some diagnoses identify another valid location.
Caveat: These scores measure localization against annotated historical causes, not successful repairs or safe autonomous handling of live incidents. A small repair pilot measures whether patches touch causal code, not whether they correctly fix the bugs.