Byren Cheema gave Jev every legal chess move as a choice, with a plain-English description of the position and each move. That guarantees legal output. It does not supply the lookahead needed to choose a strong move.
What Cheema measured
- Jev played 50 games against each of seven Maia/Stockfish settings, 50 against random moves, 50 more against Maia with the starting prompt, and 20 against itself. It also attempted 1,000 held-out Lichess puzzles.
- Against Maia and Stockfish it won zero games; the few points came from draws. Cheema estimates roughly 600–800 Elo, with wide uncertainty and different rating scales. It solved 244/1,000 puzzles.
- Against random play it won 23, drew 26, and lost one of 50 games. Repetition produced many draws despite favorable positions.
- Cheema reports a 121 ms median over 10,376 moves and $0.96 for the entire project. These are his measurements, not independently reproduced here.
Why the choice failed
- If “check” appeared among Jev’s top five options, it played that move 70% of the time. Cheema’s Stockfish analysis found checks lost more position on average than other moves. The word was a seductive clue, not evidence of a good continuation.
- It found every mate-in-one because the correct option was explicitly labeled “checkmate.” Removing those puzzles lowered Cheema’s estimated puzzle rating.
- Prompt experiments used a separate 300-puzzle practice set; replacing a drawn board with piece locations and describing moves in plain language helped modestly. A grandmaster persona did worse.
The chess experiment illustrates the limit of typed output: a move selected from valid options can still be a bad decision. Cheema shares the code and game/puzzle results; this capture reads his report, not an independent replay of the benchmark.