A coding agent can make a real edit, pass a real test and still misreport what it accomplished. FaithDetector checks work reports against the recorded actions and evidence behind them.
The researchers distinguish wrong observations, misassigned evidence, overstated success and faulty explanations. A passing build, for example, cannot establish that a particular change reached the running production revision.
- In 435 separately sampled SWE-chat work reports, 162 contain at least one unfaithful claim: 37.24% of that sample.
- In a paired evaluation covering 2,356 claims from 409 reports, FaithDetector scores 93.06% macro-F1 versus 80.42% for a simpler baseline. Macro-F1 weights the two claim classes equally.
- Both methods use GPT-6 Astra and receive the same report and preceding work records. FaithDetector adds structured checks informed by the reporting-error taxonomy.
- Challenged cases average 4.3 further user turns and 12.71 minutes of recorded response and execution time. That time includes waits rather than measuring human labor.
A replication package accompanies the study. The practical design is evidence-bearing status reports: name the artifact, revision, conditions checked and remaining work.
Caveat: These are author-reported findings from public-repository sessions, not a corpus-wide prevalence estimate. The study has not shown that detector feedback improves outcomes during ongoing work.