Yujie Feng and colleagues study agents whose model weights and reusable instruction libraries improve together during training. Their CODA audit asks whether those gains survive when someone changes the instructions while leaving the model fixed.
The authors test paraphrases, incomplete procedures and misleading instructions. They keep the selected skills fixed, isolating changed content from changes in which skills are found.
- Across four training methods and six settings, success falls by 4.9–12.6 percentage points from the intact library to the worst tested content change.
- Those losses consume 52–93% of the improvement over the same model with no skills. The changed libraries still outperform that no-skill baseline in every setting.
- Rewording alone reduces ReSkill’s ScienceWorld score from 50.3% to 39.4%. Paraphrases change outcomes on roughly 13–23% of tasks, versus 6–8% between unchanged runs.
- The authors’ separate audit finds no meaning changes in 190 paraphrased skills across four settings. A human checks half, alongside two model judges.
A model can learn to depend on the wording of its own instruction library. Treat a skill edit as a behavioral change: repeat known tasks and compare individual outcomes. An unchanged average can conceal different tasks breaking and recovering.
Caveat: These are author-run experiments on jointly trained models and libraries, not measurements of ordinary Hermes skill edits. CODA diagnoses the problem without testing a fix.