Ehsan Saleh and colleagues test what an assistant can still reveal after its original memory record is deleted. AfterDeleteBench holds model weights fixed and controls the external memory. Each synthetic secret gets an exposed-then-deleted world and a matched world that never received it.
The primary memory contains summaries that copy source records verbatim. A later experiment removes the complete secret string but keeps its components in separate records, with a public rule for assembling them.
- Deleting the source leaves a recovery route. One question recovered secrets from Qwen2.5-7B-Instruct in 68% and 64% of two 200-profile test sets. Neither never-exposed control recovered any.
- A complete surviving copy is unnecessary. With separate components, recovery ranged from 61.5% to 99.0% across Qwen, Mistral and Phi-3. Each model and test-set condition used 104 paired profiles.
- Removing a component stops complete reconstruction, but leaves partial leakage. When the number was missing, full-code recovery fell to zero while models still disclosed retained components.
- Blocking another write leaves existing information available. Removing and rebuilding every tracked dependent summary eliminated observed recovery. An exact-string write filter blocked reinsertion without clearing the remaining memory.
The study separately measures disclosure, writing a recovered secret back, and recall in a fresh session. In matched Qwen tests, writing increased later recall, but recall also occurred without it. A write filter therefore cannot substitute for propagating deletion through existing summaries.
The scoring interface introduces another trap. A legacy prompt requested a nested answer while the parser expected a top-level field. Mistral and Phi-3 scored zero recovery in one condition even though 77/104 and 78/104 raw replies contained the secret. A failed structured-output check can conceal a disclosure.
Caveat: These are author-reported tests with synthetic secrets, deliberately constructed memories and complete dependency tracking; they establish neither production failure rates nor general-purpose forgetting.
The authors plan to release code after venue acceptance. The engineering lesson is to test what a fresh session can reveal, alongside checking that the requested record disappeared.