Sebastian-Antonio Toma and colleagues build FEB-30 around 30 sequences of changing facts. They generate documents from a known sequence, then inspect the saved memory after each document arrives. They also test whether an agent can answer questions from that memory.
- The test covers updates, conflicting authors, synthesis and retrieval across 147 documents. A later document contributes new information while the system must retain earlier facts.
- In the authors’ main configuration, stored-fact accuracy averages 60.6%, while answer accuracy averages 48.8% across 11 runs.
- The average absolute gap between the two scores is 16.8 percentage points. About 44% of comparisons differ by more than 15 points; roughly one in five gives opposite pass/fail verdicts.
- Choosing the correct memory operation, such as updating versus reinforcing a fact, succeeds only 20.1% of the time. That diagnostic remains hidden inside an answer-only score.
Inspecting saved knowledge and testing the final answer catch different problems. A model may reconstruct a good answer from poorly organized fragments, while retrieval can miss facts that were correctly stored. Both checks belong in a memory evaluation.
Caveat: the results come from variants of one implementation with model-based judges and an update-favoring prompt. The announced code repository returns 404. Its “model compensation” category actually scores stored facts higher than answers, complicating that explanation.