Armature, a firm that helps dev-tool companies get adopted, ran what it bills as the largest experiment yet into how AI coding agents pick third-party services. They watched 16,893 sessions across Claude Code, Codex, and Cursor — 1,163 prompt variations on 75 realistic codebases, with agents actually installing the winner rather than just recommending one. A simulated human sat in the loop to make conversations realistic, and the full session traces are public.

The standout finding: these agents disagree a lot more than people assume.

  • All three agents picked the same tool in only 42% of cases. In one category, Claude Code chose Twilio, Codex chose OpenAI’s Realtime API, and Cursor chose Vapi
  • Codex web-searches in 94% of sessions; Cursor in about two-thirds; Claude Code leans on its own priors (~30% search) but browses 3x more pages when it does
  • Claude Code builds its own in-house solution nearly twice as often as the others (19% vs 10%)
  • Repository context dominates: the same email task produced four different winners — Resend on TypeScript, SendGrid on Python, Postmark on Go, Azure on Java
  • Being mentioned is not winning: PayPal was cited 139 times and never picked; LangChain was the most-cited framework (194 mentions) but won 4 sessions; Supabase was the most-mentioned database yet lost to Neon

Small details on vendor pages measurably flip choices — Mailgun lost to Postmark over a “1-day retention” line in its free plan, and Supabase’s bundled auth-and-storage features read as bloat to an agent that only wanted a database.

The stakes explain why this matters. Vercel reports that over 30% of deployments are now initiated by coding agents, up 1000% in six months — the buyer of developer tools is increasingly the agent itself. For engineers, the lesson is that an agent’s pick is sensitive to prompt wording, codebase language, and which model you happen to be using, so its “best choice” deserves the same skepticism you would give a confident junior. Armature discloses that it sells growth services to dev tools, and it published the full traces precisely so the results can be checked rather than taken on faith.