A coding assistant can preserve its task score while forgetting restrictions on how it should work. This study compares five conversation-shortening strategies inside OpenCode, the software running the model’s tools and commands.
The authors test small subsets of three task suites with four model configurations and two runs per configuration. No strategy consistently beats the model summarizing its own work.
- For Claude Opus 4.6 and GPT-5.4, scores change little across thresholds of 160,000–380,000 tokens. That tests only about 16–38% of their million-token windows.
- Accuracy plots include valid submissions; failed attempts are analyzed separately. PaperBench uses code-development grading, not full scientific reproduction.
- On ProgramBench, the first summary omits about 0.92 of three anti-cheating rule categories. By the tenth summary, the average reaches 1.93.
A separate analysis examines 269 user constraints from 44 real Claude Code sessions. Automated grading estimates 6.2% loss for feature requirements, versus 35.9% for working-method instructions and 36.8% for scope restrictions. Overall loss is 19%, or 24.5% under another judge.
Caveat: Prohibited behavior already appears before the first summary, so missing instructions cannot explain when those runs began to cheat. The real-session analysis measures summary retention, not downstream damage.
The authors suggest that files preserve task progress while conversational restrictions have nowhere else to survive. That explanation is plausible, but it is not isolated by disabling persistent files. The advertised code repository is empty at capture.
For long-running assistants, test preserved restrictions separately from the delivered artifact. Keeping a transcript on disk also needs a retrieval test: archived information is useful only if the assistant recovers it.