Quan Xue and colleagues test a blind spot in evaluating agent memory. A model can match the answer from a complete history even when its visible history cannot establish that answer. An agent is a model running a multi-step task with tools.
The researchers construct synthetic device logs and verify pairs of legal histories that give the reader identical text but require different answers. They also identify partial histories that do determine the answer. Cases without a proof either way stay unclassified.
- On 203 insufficient raw-prefix cases, Qwen3-32B matched 87 hidden references and abstained seven times. DeepSeek V4 Flash matched 85 and abstained 21 times on the same cases.
- Each reader received 1,600 requests across 80 histories and five context conditions. DeepSeek reused Qwen’s inputs, so this is a second reader configuration rather than an independent domain test.
- A cutoff-aware deterministic program solved every certified case. It shared its parser and rules with the certifier, making this a consistency check rather than independent validation.
- A model judge scored 1,908 of 1,920 constructed cases correctly, but still made support errors and missed the full reliability gate. A deterministic program matched the hybrid judge’s perfect sample score on the fixed answer grammar.
The study also compared compact CSV with raw logs. Its primary estimates were inconclusive for both readers; they do not establish a formatting gain or equivalence.
Caveat: These are author-reported experiments on short synthetic logs, a fixed question grammar and two hosted model configurations. They do not estimate failure rates in deployed memory systems or establish that the method transfers to natural language.