Zedong Peng compares extracted memories with coding agents that search raw conversation history. The study includes store-construction costs, which can outweigh cheaper answers later.
It tests seven methods with three models on one LoCoMo conversation: 419 turns, 152 questions and one accepted run per combination.
- With Claude Opus 5.5, Pi scored 99% for $6.95. Reading the full history scored 95% for $17.84.
- Building Mem0’s Opus store alone cost $25.88. The projected break-even against full context requires 596 questions on that same history.
- Stronger readers struggled to recover facts omitted by weaker memory writers.
- Agents received photo captions withheld from memory methods. On the 88 questions without photo evidence, Pi scored 100% and full context 99%.
The practical comparison is raw-history search against extracted memory, with ingestion metered separately. Most of the apparent accuracy lead comes from unequal evidence.
Caveat: One conversation fits every model’s context window, and a GPT-4o-mini judge grades the answers. The results cannot establish that search replaces memory generally.