Cole Medin compares model choices inside his AI Software Factory, using an 18-minute tour of three game-building runs and a separate mixed-model experiment.

The failure is in keeping the whole workflow on track

  • Medin finds open models competitive on tightly scoped tasks, but less reliable across planning, implementation, review and repeated fixes.
  • His example is a model trying to change a read-only product requirements document despite explicit rules.
  • Repeated review failures consume tokens, time and owner attention even when the model’s token price is low.

Three runs, different amounts of finished work

  • Each factory received the same starting artwork and a simplified Northstar game to build across 12 tickets.
  • Claude used Opus 5.5 for planning and review, with Sonnet 5.5 implementing; it merged 12 tickets in about 14 hours.
  • OpenAI used GPT-6 Astra and GPT-6.1 Sol; it merged 11 tickets in about 28 hours.
  • Kimi K3.1 Pro filled every role; it merged five tickets in about 44 hours and never reached the between-raid upgrade shop.
  • These are Medin’s runs in his own workflow, with visibly unfinished prototypes and no repeated-trial comparison.

Compare cost with completed work

  • Medin reports API-equivalent totals of $255 for Claude, $635 for OpenAI and $180 for Kimi.
  • He used subscriptions for the work; those totals are calculated API costs, rather than subscription bills paid for each run.
  • His scorecard reports fewer owner interventions for Claude and blind agent-review scores of 8/10, 7/10 and 6/10 respectively.
  • The same scorecard shows only 15/33, 19/33 and 9/34 end-to-end tests passing; merged tickets still left substantial failures.

Keep the strongest model on planning and review

  • Medin’s current approach uses a frontier model to absorb repository context and rules, then gives an open model a narrower implementation plan.
  • A separate earlier comparison covers 12 issues; its displayed summary reports roughly 98% quality at roughly 40% cost for Opus planning plus Kimi implementation on complex work.
  • Treat those figures as his task-specific results; the video supplies neither a reproducible evaluation protocol nor evidence that every repository will see those savings.
  • Keep the review standard and measure rework, completed features and human intervention before replacing a whole workflow.

“Just because a model is cheaper doesn’t mean that it’s actually cheaper per task.” — Cole Medin, 13:59

Caveat: This is a creator-run comparison of model combinations inside one factory. TripleTen sponsors the video; the separate mixed-model result uses an earlier experiment.