Adibvafa Fallahpour and colleagues train Memorilla to read a stored document collection and pass sixteen memory tokens to a language model. These are numerical representations inserted directly into the model’s input, carrying information selected for the current question. The answering model’s weights stay fixed.
- Read broadly, answer compactly. Each document is embedded once as a reusable vector. A trained module reads all those vectors for each question, rather than selecting a few passages to paste into the prompt.
- Better results on some tasks. With Qwen3-8B answering, the authors report 84.33% accuracy on FactKG’s 9,041 fact-checking questions, versus 80.43% with full context. Document input shrinks from 1,727 tokens to sixteen.
- Compression has losses. On PubMedQA’s hundred biomedical questions, accuracy falls to 59%, versus 67% with full context. Retrieved text also does better on some questions requiring exact names.
- Agent gains are a separate experiment. Eight learned vectors raise average TextWorld game success from 18.6% to 24.2%. Every condition retains its full textual history; the vectors supplement that history.
The memory interface requires training and direct access to the answering model’s input embeddings. The main experiment trains roughly 194 million parameters, including on the evaluation domains’ training splits. This is a model component for builders to investigate, rather than a prompting recipe for a hosted chat API.
Caveat: These are author-reported Qwen results, and the sixteen-token budget excludes the persistent document store, question and answer. The latest version read is v3, revised November 3.