Zach Lloyd, Warp’s founder and former Google Docs engineering lead, explains a measured approach to cloud coding agents in this 18-minute AI Engineer talk.
Define the system in code
- Record agents, triggers, repositories, policies, tools, execution environments, and workflow stages in a versioned definition.
- Branch and roll back that definition as you would other infrastructure.
- Keep responsibility for solving customer problems alongside responsibility for improving the agent system.
Keep the evidence from every run
- Retain conversations, code changes, screenshots, and verification videos as material for diagnosing failures.
- Use sandboxed compute and make models and harnesses, the software running an agent’s tools, replaceable.
- Connect orchestration to existing work surfaces such as Slack, Linear, Jira, and GitHub.
Measure quality and human effort
- Track cycle time, throughput, cost, and human interactions per pull request.
- Run online evaluations, tests over recorded sessions, to find specific failure patterns.
- Lloyd’s example flags redundant tests, then investigates why the agent keeps adding them.
Propose improvements, then replay real tasks
- Have an observer agent cluster failed runs and propose a pull request against the system’s definition.
- Build a benchmark from actual organizational tasks and replay them under different configurations.
- Compare model, harness, and skill changes against both quality and cost before adopting them.
- Caveat: Warp supplies the demo and reports its own improvements; the talk provides no numerical before-and-after study.
“The only way to answer those questions for real is to test on your own data.” — Zach Lloyd, 16:07