Zach Lloyd, founder and CEO of Warp, presents his company’s approach to cloud software factories at AI Engineer. The demo uses Warp’s commercial product.
Define the workflow
- Lloyd argues that sandboxed agents and centralized configuration can improve visibility and control compared with separate laptop setups.
- Store workflow definitions, agent settings, skills, tools, benchmarks and execution environments in a version-controlled repository.
- Keep model and agent-harness choices flexible, and consider who controls the data, compute and inference.
- Engineers submit work through existing tools; an orchestrator routes it through steps such as triage, specifications, implementation and review.
Measure and improve
- Record agent runs and artifacts alongside spending and cycle-time metrics.
- Judge models score runs for specific failure modes, such as redundant tests; another agent groups failures and proposes changes to the factory configuration.
- Lloyd shows a proposed pull request, not evidence that autonomous changes reliably improve outcomes.
- Replay representative past tasks with different models to compare cost and quality on your own code. The demonstrated correctness scores come from model judges; Lloyd also recommends human grading.
- Curate benchmark tasks across difficulty levels and keep independent checks: a judge’s score is not proof of correctness.
Verification and cost
- Lloyd recommends testing against specifications and gathering evidence through computer-use verification, though he does not demonstrate the latter.
- Measure cost per pull request and separate out scoring and benchmarking costs before extrapolating a budget.
- Warp Factories is proprietary; its vendor’s demo is a proposal for a workflow, not an independent evaluation of its security or productivity.
“Really set up something that you can measure and improve.”