Pete Johnson built a Pac-Man harness to test whether a faster decision can be more useful when the world keeps moving. Three interchangeable planners share the same tactical loop; one uses ordinary code, one Jev, and one Claude Haiku.
The experiment
- The tactical loop chooses turns toward a goal. Each planner selects among 11 goals, including clearing a quadrant, hunting, and escaping.
- Johnson reports 100 runs per planner. His code baseline cleared level one in 51%, Jev in 27%, and Haiku in 10%.
- He reports median missed frames per decision of 0, 21, and 59, respectively. Mean scores followed the same ordering: 5,100, 4,499, and 3,406.
- Johnson suspects latency explains much of the gap between the model planners. The comparison does not isolate latency from differences in decisions, prompts, or integration.
What carries over
A decision has a deadline as well as an accuracy requirement. Test a simple deterministic baseline and make planners replaceable behind an interface. For any real workflow, measure outcomes with its own state, timing, and error costs.
A typed answer can still be wrong, and confidence alone does not establish calibration. Johnson’s game is a narrow experiment, not evidence that the same ranking holds in enterprise systems.
Verification: read the complete article and its reported table. The experiment, its figures, and the latency explanation were not independently reproduced.