Learning Robot Actions from Gameplay

Pim de Witte of General Intuition describes a 19-minute approach to action models, which continuously turn visual observations into control inputs.

Use gameplay with action labels

Medal’s recording platform supplies videos paired with keyboard, mouse or controller actions.
de Witte reports more than 1.5 trillion action tokens monthly; this is company-reported data volume.
The shared interface is a controller: an action model can issue commands while existing robot controls handle lower-level mechanics.
Semantic labels connect a control input with its meaning, such as a greeting action.

The model receives frames continuously and emits actions as the scene changes.
de Witte contrasts that control loop with turn-based text interaction, using Olaf reassembling while rolling downhill as an analogy.
Action-dense gameplay supplies frequent corrections; he argues that this helps with tasks such as steering a drone.
World models can generate additional environments for training and exploration.

A separate model still handles long-term orchestration; the action model supplies responsive execution.
de Witte claims one model plays games and controls more than ten robot types, but withholds the release demonstration during the livestream.
Early access and a late-November or early-December release are plans stated in the talk, not a verified public release.
Proposed uses include interface testing and shared control for players who can operate a mouse but need keyboard assistance.

In the Q&A, de Witte contrasts explicit action labels with inferring actions from video alone.
A plane’s rudder command can be hard to recover from frames; recording the command avoids that ambiguity.
The talk supplies an architectural argument and vendor claims, without a public cross-robot benchmark or safety evaluation.

“You send frames and you get actions.” — Pim de Witte, 5:17