Sam Bancroft of Fuzzy Labs asks how much energy a correct model decision costs. His longer methods article distinguishes measured benchmark outcomes from estimates of electricity use.
What he compared
- Bancroft reports Jev got 200 of 231 public JevBench decisions right. He ran Jev and used JevBench’s published per-item results for other systems.
- He estimates about 30 joules per correct Jev decision, compared with roughly 150–450 joules for the general-purpose models he discusses. These are modeled estimates, not meter readings.
- A retrospective routing calculation sends hard task types to GPT-5.6 Luna and the rest to Jev. Bancroft reports 97.0% accuracy against 97.4% for Luna alone, at under half the cost. This is a re-scoring of existing results, not a tested live router.
Why the energy estimates are fragile
For closed models, including Jev, Bancroft converts published prices into energy using a ratio derived from open models. Margins and subsidies can break that relationship. The estimate therefore tracks assumptions about pricing as well as computation.
For open models, he estimates energy from active parameters and token counts. Training, hardware manufacture, networking, and grid carbon intensity are excluded. Typed decisions can be wrong, and this benchmark does not establish accuracy or energy use on another workload.
Verification: read both the shorter article and the full methods. The benchmark, routing calculation, and energy figures are the author’s and were not independently reproduced.