Jens Lehmann and Sahar Vahdati let models build and twice revise their own harnesses: the software that runs their tools, files and commands. Improvements on familiar development games often overstate gains elsewhere.
- The study follows 27 independent lineages. Every frozen version plays five development games and twenty games excluded from revision feedback.
- Mean development improvement is 8.42 points. Held-out improvement is 2.80 points, with a 95% interval from −1.44 to 7.35. The score combines completed levels with action efficiency against human baselines.
- Nine final harnesses become cheaper and better; nine become cheaper and worse. Lower spending alone is an ambiguous success metric.
- Repeating one unchanged harness ten times yields a score standard deviation of 3.99 points, warning against overinterpreting small single-run changes.
Preserve the working baseline and test capability separately from whether an edit executes successfully. The paper releases source snapshots, outcome records and analysis scripts for auditing its results.
Caveat: One public game suite and two revisions cannot establish a general law of self-improvement. Training exposure and execution variation remain possible.