Imene Kerboua and colleagues introduce WebArena-Pro, a fixed set of browser tasks spanning twenty self-hosted applications. Each task starts from reset application state. That makes failed workflows easier to reproduce than tests on changing production websites.

The released code separates 100 development tasks from 300 test tasks. Applications include Nextcloud, Mattermost, Immich and Penpot, plus custom sites modeled on familiar services. They reproduce workflow patterns, not every feature of the reference products.

  • The test set includes 96 cross-application tasks and 79 audio/video tasks. Those groups can overlap.
  • Agents, models running multi-step tasks with tools, can request transcripts and sampled video frames. Some configurations instead accept native audio/video input.
  • At a 50-action limit, the authors report 62.3% success for both Codex/GPT 6.1 Sol and Claude Code/Opus 5.5. OpenCode/Kimi 3.1 Pro reaches 60.7%.
  • Claude Code completes 55.2% of cross-application tasks, compared with 65.7% of single-application tasks. Preparing a document or order often fails to produce the requested saved result.

Media access helps, but it is not the whole explanation. On a post-hoc subset of 43 tasks judged to require media content, Opus scores 48.8% with media tools. It still scores 14.0% without them. Metadata, other clues and guesses can contribute to scores.

Caveat: These results compare complete model-and-tool setups, with partly model-based grading; they cannot isolate model capability or wrapper quality. Equal action limits also leave tokens, native tools and time unequal. The reported runs are author evaluations, not independent replication.

The paper’s abstract and results table give Kimi 60.7%, while one results paragraph says 60.3%. The table value is used here. The arXiv record lists only v1.

Daily coverage