Tural Mehtiyev and colleagues test what happens when project instructions fall behind the code. Coding assistants usually repair the requested bug, but leave the misleading instructions for the next session.

The study compares stale, corrected and deleted instructions across 22 maintenance tasks and 732 runs of Haiku 5.5 and Sonnet 5.5. Its three real cases concern a test command, a code location and a component-creation procedure.

  • Agents use the obsolete test path in 78 of 80 stale-command runs. Piping output through head or tail hides the failing exit status.
  • In 19 of Sonnet’s 40 stale-command runs, its first check after the fix executes no tests. All 40 final patches still pass the researchers’ restored tests.
  • Sonnet uses 7.8 calls to tools with the stale command versus 5.8 with the corrected one, averaging per-task medians. Deleting the instruction also gives 7.8.
  • Agents recover from 130 of 133 obsolete-path encounters, but only 21 final responses identify the mismatch. None of the 732 runs edits an instruction file.

Correcting the instructions eliminates obsolete-path operations. Deleting them leaves some agents guessing the same wrong paths. The replication package contains the authors’ data and analysis code.

Caveat: These runs repeat three cases, and the exploratory extra-call finding loses statistical significance after correcting for multiple comparisons.

The tasks requested code fixes, not documentation maintenance. The practical implication is to check documented commands and actual test counts separately from whether the final patch passes. Proposed safeguards, including explicit mismatch reporting, were not evaluated here.

Daily coverage