Panrong Tong and colleagues at Alibaba describe MAI-Gym, a platform for training agents that operate phones, computers, and browsers. Evaluation and training share the same interaction loop, so changes to prompts, screenshots, or action handling reach both together.
Its task-building pipeline starts with connected files and app records, then defines the requested outcome before generating the code that checks it:
- Separate generation calls write code that creates a completed state and code that verifies it. Checks must reject the initial state, accept completion, and reject the reset state again.
- The authors’ nine-billion-parameter model scores 78.4% on MobileWorld and 76.2% on OSWorld-Verified. These are unseen tasks in environments used for training; supervised warm-up supplies most of the overall improvement.
- Training only in a train-ticketing clone raises success on 52 paired tasks in the real app from 28.8% to 48.1%. Clone success rises further, to 73.1%, so the simulation gap grows.
- Unfamiliar desktop work remains difficult: 3.7% binary success on OSWorld 2.0, versus the much stronger headline result on OSWorld-Verified.
The requirements-first checks and shared execution code are useful engineering patterns. The reported training run used 512 GPUs for approximately 96 hours; a small model does not imply a small training budget.
Caveat: results are author-reported, and the public repository still says the framework source is coming soon despite the paper’s release claims.