Zach Lloyd, Warp’s founder and former Google Docs engineering lead, explains a measured approach to cloud coding agents in this 18-minute AI Engineer talk.

Define the system in code

  • Record agents, triggers, repositories, policies, tools, execution environments, and workflow stages in a versioned definition.
  • Branch and roll back that definition as you would other infrastructure.
  • Keep responsibility for solving customer problems alongside responsibility for improving the agent system.

Keep the evidence from every run

  • Retain conversations, code changes, screenshots, and verification videos as material for diagnosing failures.
  • Use sandboxed compute and make models and harnesses, the software running an agent’s tools, replaceable.
  • Connect orchestration to existing work surfaces such as Slack, Linear, Jira, and GitHub.

Measure quality and human effort

  • Track cycle time, throughput, cost, and human interactions per pull request.
  • Run online evaluations, tests over recorded sessions, to find specific failure patterns.
  • Lloyd’s example flags redundant tests, then investigates why the agent keeps adding them.

Propose improvements, then replay real tasks

  • Have an observer agent cluster failed runs and propose a pull request against the system’s definition.
  • Build a benchmark from actual organizational tasks and replay them under different configurations.
  • Compare model, harness, and skill changes against both quality and cost before adopting them.
  • Caveat: Warp supplies the demo and reports its own improvements; the talk provides no numerical before-and-after study.

“The only way to answer those questions for real is to test on your own data.” — Zach Lloyd, 16:07