Nikhil Kumar Jha and Amitoj Cheema rerun LoCoMo, a fixed set of conversational-memory questions, while changing answer instructions and grading models. Their results show how much an apparently better memory score can depend on the surrounding test.
They compare their Becos memory system with the complete conversation under matching settings: ten conversations, 1,540 questions, two answering models, two prompts and three graders. A judge model is a model grading another model’s output.
- Grading changes the result: Switching only the judge moves a score by up to 7.2 percentage points, the authors report. Changing answer instructions moves it by up to 9.3 points.
- The answering model matters: With short answers and the primary judge, Becos scores 79.7% versus 73.0% for the full conversation using GPT-4o mini. With GPT-4.1 mini, they nearly tie at 83.2% versus 83.1%.
- Less text is a separate benefit: Becos supplies 90.1% fewer context characters on average. The reported savings exclude the resources needed to store and search memories.
- The baseline needs scrutiny: Adding a date to every message improves the full-conversation score from 67.3% to 73.3% on one conversation. The effect across all ten is unknown.
- Longer answers confound the prompt comparison: The seven-step prompt adds reasoning instructions, removes the short-answer limit and grades only the final answer. Those changes are bundled together.
The release includes answers, verdicts, scoring code and a small author-run human check. It supports rescoring and baseline reruns, but omits Becos’s implementation and retrieved contexts.
Caveat: Both authors work for Becos’s developer, and its research configuration was developed on LoCoMo without a fully unseen test set. These personal-chat results do not establish coding-work performance.
A useful memory comparison needs the same answer model, prompt, grader and baseline layout. Shorter retrieved context can save answering resources even when it merely matches full-context accuracy.