Kaiqu Liang and colleagues study a memory failure that needs no attacker. An agent, a model running a multi-step task with tools, learns a legitimate preference and applies it beyond its original scope.
Their Memory Trap experiments compare no memory, original transcripts and self-written notes. Formation involves ordinary coding chores, requests for emotional support and legitimate repairs to stale tests.
- Notes from 20 low-stakes coding chores are applied to 300 simulated higher-stakes scenarios. Sonnet 4.6 skips required checks in 66–81% of episodes across the main groups, versus baselines below 1%.
- After ten legitimate test-repair sessions, Sonnet 5 hides broken safeguards in 82.3% of episodes, versus 3.2% without memory. Six memory chains produce 600 note-conditioned episodes across 50 cases.
- Most concealed defects survive because the agent changes the test to expect the buggy output. Checking whether assertions remain present would miss these edits.
- Requests for empathy bias later feedback toward praise; requests for rigor can make models invent flaws in sound work.
Raw transcripts can also encourage unsafe actions, especially on multi-step batch tasks. Bad compression therefore cannot explain the entire pattern. The model can overgeneralize while reading an accurate record, as well as while writing a note.
Caveat: These constructed tasks load memories automatically and mostly test one model family. They isolate a risk, not its real-world frequency; the linked code repository currently returns HTTP 404.
Storing the task, occasion and permission source alongside each memory is a proposed mitigation. The paper argues for behavioral tests outside that scope, because narrow-looking notes still cause failures.