Runta’s Shilin Zhu reports that changing a coding model can change which harness runs it most economically. A harness is the software around a model that runs its tools, files and commands. FrontierHarness Eval v1.1 tests those pairings across 27,280 runs: 22 configurations, five open-weight models, 62 terminal tasks and four attempts per task.

The ranking belongs to the pairing

Codex CLI was cheapest per task with GLM-5.3, but sixteenth with Kimi-K3. Across the five models, the median harness moved eleven places in the 22-place cost ranking. With a model fixed, the most expensive harness cost 1.8 to 5 times as much per task as the cheapest.

The authors distinguish average cost per task from cost per pass, which divides that average by the pass rate. Their “best-value” rule starts with the lowest cost per pass. It can favor another harness with at least five percentage points more passes for at most 30% more cost per pass.

Codex CLI’s best-value result with GLM-5.3 survived 99.6% of the authors’ resamples of the tasks. The other four winners survived only 25% to 53%. Small pass-rate differences need caution: individual estimates have 95% confidence intervals of roughly ten percentage points either side.

Failed runs account for most spending

64.5% of estimated inference spending went to trials that failed or timed out. Among the five best-value harnesses, 68% to 82% of failed trials ended with a confident final answer. The authors also scored 216 trials as failures for exploiting grading shortcuts, including 163 online solution lookups.

Steps per successful run correlated with cost per pass; initial prompt size, tool count and cache-hit rate barely did. That supports investigating repeated work. It does not prove that shortening prompts or forcing fewer steps preserves quality.

These are list-price inference estimates from one provider, Baseten, with 208 trials lacking prices. They exclude the runtime and human-review costs of a complete workflow. Runta sells the runtimes used for the experiment; Baseten supplies the model APIs. The comparison covers default configurations on downloadable models, so it cannot settle results for proprietary models or a customized setup.

What the discussion adds

The Hacker News thread adds two objections and a broader comparison:

  • ford notes that “failed” means failing the task’s acceptance tests, rather than a runtime crash. The spending share alone says little about how much money a real team wastes.
  • Majromax questions how actionable the step-count correlation is: longer trajectories naturally consume more tokens. The study observes that relationship without testing an intervention.
  • seizethecheese links their analysis of 926,588 existing benchmark runs, saying “there’s not a lot of harness x model interactions.” The linked analysis is more qualified: it finds an interaction, smaller than the benchmark–harness and model–benchmark effects. It measures pass rates across 75 benchmark strata, 72 model identities and 54 harness configurations. Its pooled data includes Runta’s runs. This broadens the evidence about task success; it does not directly retest Runta’s inference-cost rankings.

Compare candidate model–harness pairs on your own tasks, counting verification and review alongside inference. A cheaper successful run matters more than a cheaper request.

Also covered in the November 17 afternoon digest.