Hightouch’s Albert Jin, Neil Desai and Delaney Mackenzie test cheap models that flag customer profiles combining different people’s records. Jev plus Claude Opus 5.5 nearly matches Opus alone by sending uncertain cases for a second opinion. The deciding factor is whether confidence scores identify the mistakes worth escalating.

  • First test: 1,000 synthetic profiles, half valid and half incorrectly merged. Routing Jev’s 30–90% confidence cases to Opus sends 22% for review and reaches 98.8% accuracy at $0.88 per thousand profiles. Opus alone reaches 99.0% at $3.86.
  • Harder cases: The same routing rule reaches 95.3% at $2.94 per thousand, versus Opus’s 95.1% at $5.39. The headline fourfold saving becomes less than twofold.
  • A stronger first score can mislead: GPT-6 Luna does better alone on the easier test. Its confident mistakes escape escalation, leaving Luna plus Opus at 91.6% on the harder cases.

Caveat: These are Hightouch’s synthetic tests with tuned thresholds, not measured production savings. Real error rates, model prices and the cost of missed mistakes can change the choice.

The 50-comment thread on Hacker News adds practitioner counterexamples and an untested alternative.

What the thread adds

  • netshade — reports that a newer cheap model erased the advantage for their workload: “introducing a decider layer didn’t offer huge wins.” Browsing delays also overwhelmed the decision model’s speed.
  • gbrindisi — reports a similar security-review setup “working well after initial calibrations.” This is an anecdote about tool-call and intent checks, without a published accuracy comparison.
  • nico — cautions that the study is “not making claims of, or demonstrating general usage improvements or general coding agent improvements”.

Does the fallback have to be Opus? giancarlostoro asks: “What about Sonnet? Haiku? Does it need Opus specifically?” The article does not test those alternatives. A cheaper fallback could change the calculation again.

HN handles are pseudonymous and no per-comment scores are public; HN’s ranking supplies the sampled order, not a consensus.

Daily coverage

November 6 Afternoon covered the study’s routing tables and harder-case comparison.