Oleeviya Babu Poikarayil and colleagues test a failure that retrieval scores can hide: an assistant finds a correction, then answers from an obsolete memory anyway. Their synthetic histories make later numbered statements replace earlier ones.
A controlled comparison keeps four statements per question and the same prompt. It replaces one unrelated statement with the obsolete competing fact while leaving the latest fact visible.
- Across 74 questions, Qwen2.5-7B accuracy falls from 100% to 44.6%. Qwen3-14B falls from 100% to 87.8%.
- The official classifier labels all fifty harmful answer changes stale. A supplementary audit finds mixed old/new outputs; stricter scoring preserves the comparison’s direction.
- In a separate one-hundred-question test, increasing retrieved evidence from 64 to 512 tokens lowers Qwen2.5 accuracy from 45% to 32%. The latest fact remains present in every retrieved set.
- Both versions already appear at every budget in the conflict questions. The budget effect therefore also involves added surrounding evidence.
The synthetic updates often contradict familiar world knowledge. The experiment cannot isolate chronology alone, and the same questions informed prompt development. These are author-run tests, not measured failure rates for a deployed assistant.
The released experiment materials support testing retrieval and answer freshness separately. Marking superseded facts inactive is a proposed engineering response, rather than a mitigation validated by this study.