Manish Kapur of Sonar and Addy Osmani of Anthropic discuss verification for coding agents in a 19-minute conversation on AI Engineer.
Scale review around executable evidence
- Kapur cites Anthropic merging about 30,000 changes per day; Osmani describes roughly 2,500 daily changes back in May and the review workflow behind the growth.
- Claude performs a first review, with browser interaction, end-to-end tests, linting, and other checks appropriate to the change.
- Hooks inspect edits and feed corrections into the loop; team rules in shared instruction files guide the review.
- Humans inspect the evidence, including screenshots or videos of user-facing changes, and retain ownership of the result.
Match automation to confidence
- A “lights-on” software factory keeps human judgment inside repeated planning, building, checking, and shipping loops.
- A “dark” factory shifts humans toward intent and exceptions; Osmani expects mixed workflows rather than universal full autonomy.
- Property tests, which check invariants across inputs, and end-to-end or visual tests make “verify the work” concrete.
Put different checks at different layers
- Kapur recommends deterministic code checks, agent review for logic and intent, and dependency/supply-chain checks.
- Osmani emphasizes agent identity, activity tracking, permissions, and sandbox boundaries as usage expands.
- Both favor labeling agent-written pull requests while keeping a named human accountable.
- Sonar’s survey figures describe reported practices and expectations; they do not establish that generating code is a solved engineering problem.
Caveat: the throughput numbers are attributed company claims. The talk provides no change-size distribution, incident-rate comparison, or independently verified relationship between merge volume and quality.
“I might have kicked off a Claude run to complete a piece of work, but I’m still on the hook.” — Addy Osmani, automatic-caption product spelling corrected