George Cameron, co-founder of Artificial Analysis, presents a 20-minute AI Engineer talk on model selection and the cost of multi-step agents.

Count the whole task

  • An agent’s bill combines input tokens, answer tokens, reasoning tokens, and the number of model calls needed to finish.
  • Prompt caching reuses previously processed input and can qualify those tokens for a lower price.
  • Cameron’s AA Briefcase knowledge-work benchmark example costs about $21 per task across 165 steps.
  • He puts that example above $200 with no cache hits, illustrating the source title’s roughly tenfold difference.
  • Input volume, the share of input served from cache, and turns per task dominate his cost breakdown.

Reasoning effort changes the trajectory

  • Higher reasoning settings can change both the tokens used in a call and how many calls the agent makes.
  • Conversation history carries earlier material into later inputs, so costs accumulate across the run.
  • Cameron shows the same Opus 5.5 model costing roughly $1 per task at low effort and $21 at maximum effort.
  • Measure completion quality alongside spending before treating a lower reasoning setting as a replacement.

Use public rankings to make a shortlist

  • Artificial Analysis combines ten evaluations in its Intelligence Index and also publishes task-specific results.
  • A model’s overall rank can hide strengths in particular tasks, such as navigating complex spreadsheets.
  • Cameron reports large price differences between frontier models and cheaper alternatives, including downloadable open-weight models.
  • His historical charts show a fixed benchmark capability level becoming roughly ten times cheaper over about a year.
  • Those charts describe measured capability tiers; future prices and savings remain uncertain.

Decide whether better answers still add value

  • For open-ended business analysis, Cameron argues that greater capability can keep improving the result; the decision starts with a budget.
  • For a bounded question such as last month’s subscription spending, several capable models may return the same correct total.
  • A Pareto frontier marks options that another choice cannot beat on both quality and cost.
  • Use that frontier to find the cheapest model meeting the task’s required quality, or the strongest model within the budget.

Benchmark the workload you will run

  • Shortlist models using public results, then test them with your own context, tools, and agent setup.
  • For production, Cameron recommends building a task-specific benchmark and comparing a handful of candidates.
  • Evaluate the inference provider too: price, response speed, and actual cache hit rate affect the same model’s economics.
  • He closes by pitching Optima, Artificial Analysis’s benchmark-generation product; its claimed benefits come from the vendor.

Caveat: the $21 and over-$200 figures describe Cameron’s benchmark example and uncached estimate. The talk does not establish a tenfold saving for every agent or provide a cache-configuration tutorial.

“The challenge for us as builders is making the trade-off decision.” — George Cameron, 15:43