Kush Agrawal, a Duolingo staff engineer, explains a 12-minute AI Engineer case study of generating and merging unit tests for the iOS app.

Why they started

  • Agrawal says the app has nearly two million executable lines; overall unit-test coverage began at 11%.
  • They targeted the more testable MVVM layers: repositories, view models and data sources, rather than the whole legacy app.
  • The aim was to make faster human and agent code changes safer to verify, not to maximize test-line count.

The production loop

  • Identify: each merged PR refreshes an Xcode coverage report in S3; a selector favors MVVM files by 2.5×, penalizes large files and skips UI code.
  • Write: a Temporal child workflow per file runs a coding agent, reads repository testing guidance, writes tests and opens a labeled PR.
  • Verify and resolve: a periodic workflow watches CI and PR state, applies deterministic pre-commit fixes, lets an agent try real CI failures up to five times, and assigns a human reviewer using git blame.
  • Iterate: rejected files cool down for 30 days; reviewer and automated-judge feedback improves the next batch. Engineers can request tests on their own PRs by label.

Failure modes and reported result

  • Early local batches merged 8 of 17 and 19 of 40 PRs. Agents mixed XCTest and Swift Testing, struggled with a hard-to-mock singleton and wrote timing-sensitive tests.
  • Linux runners could not build iOS code locally; moving to Macs let agents build and lint before opening a PR.
  • Guardrails reject production edits, modified or disabled existing tests, and long waits in generated-test PRs. Every merged PR still needs human approval.
  • Agrawal reports over 700 merged test PRs and about 225,000 new test lines. Overall coverage rose from 11% to 24%; targeted MVVM coverage from 9% to 59%.

Caveat: These are Duolingo’s reported internal figures; coverage measures executed lines, not whether every test catches a real defect.

“The bottleneck is really shifting from writing to verifying.”