Warp’s Ben Holmes shows how to turn coding-agent conversations into reviewed improvements to skills and configuration in a 19-minute AI Engineer talk.
Capture what the agent actually did
- Centralize conversations across the team, either from local histories or a shared execution system.
- Keep user/assistant messages, tool calls and results, and the resulting code diff or pinned pull request together.
- Successful output can hide stale instructions, failed tools, and time spent rediscovering the environment.
Score specific behaviors
- Holmes uses separate rubrics for redundant tests, verbosity, and efficiency.
- Test coverage should follow the team’s intended testing strategy; fewer tests alone are a poor objective.
- Concise comments and summaries can reduce human review friction.
- Efficiency checks need the tool trace to distinguish direct progress from repeated unsuccessful setup attempts.
Turn recurring failures into proposed changes
- Score completed sessions, retain reasons for each judgment, and periodically group failures by cause.
- Give a separate improvement agent those reports and access to the relevant skills and configuration.
- His examples repair port-discovery guidance, changelog instructions, and missing permission to spawn subagents.
- Proposed changes arrive as pull requests for the team to review.
Make the loop visible
- Holmes presents Skill Doctor as a one-time review of local coding-agent history.
- Warp Factories applies the same pattern continuously, with a review inbox for suggested changes.
- Track scores around configuration changes, then inspect the underlying cases before concluding that a change helped.
- The talk demonstrates a workflow and example repairs; it supplies no controlled before/after productivity benchmark.
“Agents that look through their past work and figure out how to improve their own setup.” — Ben Holmes, 0:18