Halv published an unusually inspectable interim benchmark of its coding-agent workflow: 14 SWE-rebench tasks, three paired repetitions per task, comparing vanilla Codex with a hierarchy in which Jev selects worker models. The result is useful evidence, but it is evidence about the whole Halv stack, not Jev in isolation.

The headline result

  • Halv reports 25 verifier passes out of 42 runs in each arm.
  • Its selected runs recorded $144.67 in model cost versus $337.50 for vanilla Codex—a 57.1% reduction by Halv’s token-based accounting.
  • Cost per verifier pass fell from a reported $13.50 to $5.79.
  • The routed workflow used 170.1 million tokens, versus 48.8 million for vanilla: 3.49× as many tokens, despite the lower recorded cost.
  • These are Halv’s measurements, not provider invoices. I reproduced the published archive’s checksums and arithmetic verification, but did not rerun the repository tasks or authenticate billing.

What Jev actually did

  • Both arms used Codex 0.155.1 with GPT-6 Astra at medium reasoning as coordinator.
  • Vanilla Astra handled the task alone.
  • In the Halv arm, Luna 6 first investigated the repository, then Jev selected the L1 worker model and reasoning effort; L1 could delegate bounded work to a cheaper L2 worker.
  • The observed routes used Sol 6 and Luna 6 at different reasoning levels. Only four of the 42 Halv runs created L2 work.
  • Jev’s typed output selected a route. That does not show that the route was correct, nor does it isolate the value of Jev from the rest of the workflow.

Where the comparison is strong

  • The runner alternated Halv and vanilla within each repetition and retained valid failed runs rather than deleting expensive failures.
  • The public archive includes 84 sanitized run summaries, per-role costs, verifier rewards, a selection rule, an attempt ledger, checksums, and a verification script.
  • The script successfully verified all 42 pairs, 14 tasks, 84 runs, session accounting, exclusions, and file hashes locally.
  • The full task table exposes regressions instead of hiding them. Halv cost more on four of 14 tasks.

The failures matter

  • On Perry-3982, Halv was 74.9% cheaper but passed only one of three runs; vanilla passed all three. Lower cost did not preserve task-level correctness.
  • Two gains on ArcadeDB-4281 and HugeGraph-3037 offset the two missing Perry passes, producing equal aggregate pass totals from different outcomes.
  • Several tasks failed in both arms. Spending less on a failed task is operationally relevant, but it is not a solved task.
  • Three repetitions per task are too few to establish stable per-task performance or statistical equivalence.

What the result does not prove

  • Halv changed Jev routing, worker models, reasoning effort, investigation, delegation, code navigation, output filtering, and context processing together.
  • The checkpoint covers the first 14 tasks of a planned 111 and stopped at the user’s request rather than at a preregistered endpoint.
  • Thirteen infrastructure-invalid attempts, ten runs from a rolled-back policy experiment, diagnostics, and unknown costs were excluded from the headline. The ledger discloses them, but the result is not the all-in campaign bill.
  • Published source hashes refer to private snapshots; prompts, transcripts, credentials, and full source artifacts are unavailable. The archive can verify internal consistency, not independently reproduce the experiment.

The defensible conclusion is narrow: this particular routed workflow recorded substantially lower model cost while matching the comparison arm’s aggregate verifier-pass count. It is promising evidence for routing cheap workers, not proof that Jev alone creates 57.1% savings or that quality was equivalent on every task.