A Redwood Research-led team gives auditors detailed records of experiments containing human-planted sabotage. One researcher produces ten honest and twelve sabotaged records across ten research questions. Twenty-five human auditors complete 104 reviews, with AI assistance and permission to run validation experiments.

The attacks can alter experimental design, omit important comparisons or distort code and presentation. A readable execution history can faithfully document a misleading experiment.

  • After 90 active minutes per review, humans fully identify 17.4% of planted mechanisms, averaging each of the 33 mechanisms equally.
  • Their ability to rank sabotaged records above honest ones scores 0.59, where 0.5 means chance. The primary 95% uncertainty interval spans 0.37–0.77.
  • Fable 5.1 fully identifies 31.8% without running experiments. Its observed advantage over humans remains statistically uncertain in the paired comparison.
  • Combining reports supplies additional diagnoses, but the reported pooling assumes someone can select the correct suggestions. It does not test a working method for resolving conflicting advice.

The saboteur knows the honest results before constructing attacks. Separate predictions made before experiments often miss their outcomes, so that advance knowledge is a substantial assumption. Retrospectively selecting completed attacks judged hard to diagnose reduces identification, but does not consistently reduce general suspicion.

The records include file changes and terminal activity, with gaps for private reasoning, coding-agent conversations and some large intermediate artifacts. Identifying a mechanism also differs from applying a fix and demonstrating a corrected conclusion; the study measures the former.

Caveat: This measures detection of human-planted sabotage under one protocol, not how often deployed AI systems attempt it. One saboteur and ten selected projects limit generalization. The paper’s research-material link is an access-request form, rather than a directly downloadable public archive.

Daily coverage