MemPABench asks whether an assistant remembers how a person wants help, including when to confirm, ask questions or initiate contact. It measures the preference selected and the behavior actually delivered.
- Four simulated users have fourteen interaction attributes across work and personal contexts. Each timeline contains 99 sessions and five preference changes.
- Across four underlying models, file memory selects the right preference 59.2% of the time, versus 49.6% without memory. Three retrieval-based systems score 49.9–50.0%.
- Applicable preferences appear in only seven of 334 inspected retrieval inputs, compared with sixty of 111 file-memory inputs. Task-related memories can crowd out interaction rules.
- File memory gets both contrasting contexts right in 14.4% of pairs, versus 21.8% without memory. A remembered personal rule can spread into work.
Check the whole chain from storage to retrieval to action. Even when supplied the correct preference, models sometimes select it and then behave differently. The project page accompanies the methods and environment.
Caveat: These results concern simulated people and specific memory integrations. They do not establish that every retrieval design performs the same way.