Rayan Krishnan of Vals AI presents a 15-minute AI Engineer talk on choosing models using the cost and quality of actual work.

Measure the task you actually buy

  • A model’s overall benchmark rank can change across finance, programming languages, libraries, and individual repositories.
  • Evaluate the model together with its harness, the software that runs its tools, files, and commands.
  • Compare cost per completed task, response time, and quality alongside the price per token.
  • In Krishnan’s tax example, the cheaper-per-token model consumes enough extra tokens to cost more while scoring lower.

Slice the results before choosing

  • In Excel modeling, leveraged-buyout tasks score lower than the full benchmark, and the best cost-quality tradeoffs change.
  • A Pareto frontier marks choices that another option cannot beat on both cost and quality.
  • Krishnan shows a model dropping off that frontier when the workload narrows to leveraged-buyout modeling.
  • Partial credit measures how much of an answer is right; fully correct answers measure how often the entire result meets the rubric.

Build tests that survive contact with agents

  • Validity: use real work and grading criteria that reflect the output people need.
  • Reliability: collect enough tasks and repeated samples to distinguish consistent performance from a lucky run.
  • Discriminability: choose tasks that separate models and leave room for future improvement.
  • Restrict access to other models and the internet when those resources would invalidate the comparison.
  • Grade whether an answer commits to a usable conclusion; listing several alternatives can game a checklist.
  • Support long-running, resumable evaluations and the clarification turns that real work requires.

Reuse the firm’s own work history

  • Krishnan says a free-access experiment used $1.5 million worth of tokens in one month across roughly 10–12 engineers.
  • Vals replayed historical GitHub work as agent tasks to compare tools, models, costs, and completion times.
  • He reports roughly 90% lower spending while maintaining output after using those results to guide tool purchases and model choices.
  • For other professions, completed work and standards may live in email, SharePoint, or data warehouses.
  • Use those tests to set usage limits, negotiate prices, route tasks, and check fine-tuning against a separate evaluation set.

Caveat: these are Vals AI’s benchmark examples and internal savings claims. The talk does not provide a controlled productivity study or a universal hosted-versus-self-hosted cost ranking.

“Agents solve the task, not the problem.” — Rayan Krishnan