Researchers at USC and AWS test what happens when reusable assistant instructions outlive their environment. A skill packages instructions, code and references. Saved paths and field names reduce repeated discovery, but become liabilities when the underlying system changes.

The study compares Opus 4.7 and Haiku 4.5 across terminal tasks, historical tax forms, application APIs and a science simulator. It measures correctness and execution effort separately.

  • With stale, highly specific instructions, Opus’s tax-form accuracy fell to 73.1%, below 85.6% without the skill. It still used fewer turns, showing how a faster run can be a worse result.
  • In the science simulation, stale instructions made Opus use 38.9% more steps than starting without the skill.
  • After maintenance, Haiku’s tax-form field accuracy rose from 30.5% to 97.5%. Maintained instructions reduced steps by up to 37% in the science task while preserving its perfect observed task score.

The proposed maintenance process records checks from successful executions, then replays them before reuse without model calls. A changed observation triggers conflict verification. A model can repair affected instructions, validate them or rebuild the skill when repair fails. Those later stages can incur model costs.

Detection has limits. Replay flagged 34 of 51 labeled terminal-task conflicts; verification supported 28. Content changes outside the recorded properties escaped detection. Two harmless changes survived verification and caused unnecessary repairs.

Maintenance also sometimes worsened held-out performance. Application-task success with one procedural skill fell from 62.7% to 56.9%, with substantial uncertainty. Validation therefore needs both quality and effort checks.

Caveat: The study evaluates controlled changes and one maintenance event, not repeated upkeep in production. Its 21–42-use token payback compares maintenance with retiring the skill; continuing stale reuse gives a different estimate. Environment computing is excluded, and automatic refresh of checks after an accepted update remains future work.

Daily coverage