CompAct-Bench measures whether an agent, a model doing multi-step work with tools, can finish after its conversation is compressed. The test grades the resumed task rather than the summary’s readability.

The researchers start with 200 tasks solved in all three screening runs. They shorten earlier conversation to roughly 10% of its character length, restore the corresponding environment and resume with a fixed executor.

  • The set contains 79 web-research tasks, 96 coding tasks and 25 internal workspace tasks. Twelve models serve as compactors.
  • The strongest compactor reaches 68.5% completion accuracy; the mean is 56.7%. Each compacted history receives three independent continuations.
  • Later coding work survives better when changes already exist in repository files. Research findings often exist only in conversation.
  • Some summaries preserve a fact but change its status, turning an unresolved lead into a conclusion or reopening a rejected candidate.

The project offers a useful evaluation pattern: test actual resumed work and preserve what remains unverified. Repository edits help, but findings that have only been located still need to survive the handoff.

Caveat: These are author-reported results under an aggressive character budget and fixed execution limits. Cross-task associations do not isolate every cause of failure, and the workspace suite is internal.

Daily coverage