Addy Osmani describes how the Claude Code team checks agent-written code: run the application, collect evidence, and test the checks themselves. The team reports that Claude writes and reviews roughly 90% of its code. Delegation depends on giving it a reliable way to find its own mistakes.
- Run complete user flows. End-to-end tests exercise the feature in the running app, with screenshots and traces showing where it failed.
- Challenge the assumptions. Property tests generate many inputs to try to break a rule. Differential tests compare old and new implementations on identical inputs.
- Keep grading rules separate. Review instructions come from the base branch, so a proposed change cannot rewrite its own checks. Reviews apply to a fixed commit.
- Check the reviewer. Replay past incidents, include stable changes to catch false alarms, and keep tuning examples separate from the final test set.
The team turns repeated misses into targeted rules and brings in a human after three unsuccessful repair rounds. New blocking checks first run in observation mode. This is a first-party process account; its separate “up to 10 points” result concerns structured-file generation, not this entire engineering workflow.
The 151-comment thread on Hacker News adds cheaper alternatives and questions about what deserves human attention.
What the thread adds
- OtherShrezzing — argues for reusable Playwright checks over repeated browser-driving model calls: “It is write-once-use-often, so has a marginal cost near zero.” That is a proposed cost advantage, not a comparison measured in the thread.
- FartyMcFarter — challenges deleting an unexplained flaky test. For tests isolated from outside dependencies, investigate “by bisecting to determine when the test became flaky.” Otherwise, identify the troublesome dependency before deciding to delete it. octoberfranklin replies that isolation still leaves variable thread timing: “Tests need to be not just hermetic but also deterministic.” Isolation and repeatable execution require separate attention.
- koolba — disputes the assumption that humans cannot review every change: “The trick is to break things into small enough pieces that you can review them.” Smaller changes offer another way to keep review manageable.
- causal — questions whether passing checks establish that the agent understood the task: “It can write perfectly good verification for the wrong problem.” d-sanoj reports a related failure: “The tests just agreed with the bugs sometimes.” These objections put domain judgment and the choice of requirements ahead of the number of checks.
- jrflo — provides a counterexample to the thread’s complaints about product reliability: “The desktop app is actually great now.” kevincox, by contrast, reports that secondary features work “only on occasion.” These are conflicting personal experiences, not a reliability survey.
HN handles are pseudonymous and no per-comment scores are available. HN’s own ranking shaped the fetched slice; this is a slice of the thread, not a consensus.