Anthus tested the general GPT-6 Luna model as a stand-in for OpenAI’s limited-preview Decisions API: one request, reasoning disabled, a strict JSON schema, and a fixed set of answers. The result is a useful warning about confidence-driven automation—but not a benchmark of the specialized API itself.

The test

  • The benchmark used 3,600 ProofWriter problems: 1,800 open-world true/false/unknown decisions and 1,800 closed-world true/false decisions.
  • Difficulty was measured by proof depth, from facts stated directly to answers requiring five chained rules.
  • Anthus says it recomputed every gold answer with an independent solver and preregistered the separate confidence probe.
  • The same prompts and options went to Luna and Jev; Luna ran with reasoning off at OpenAI’s default sampling temperature.

Accuracy fell with depth

  • Anthus reports Luna at 64.1% on the open-world task and 65.0% on the closed-world task, versus 83.8% and 89.3% for Jev.
  • At depth zero, the models were relatively close. At depth five, Luna reached 46% and 45%, while Jev reached 81% and 89%.
  • The measured gap therefore grew with the amount of chained inference rather than appearing only on obscure edge cases.
  • These are Anthus’s published benchmark results. I reviewed the article and its saved-result page; I did not rerun the 7,200 model calls.

The confidence problem is worse than simple overconfidence

  • In the separate probe, Luna assigned at least 99% confidence to 2,672 answers; only about 68% were correct.
  • Anthus reports expected calibration error of roughly 0.32 for Luna, versus 0.03–0.04 for Jev on the same tasks.
  • More importantly, Luna’s confidence AUROC fell to 0.52 and 0.50 at depth five—approximately random ordering of correct and incorrect answers.
  • A calibration curve can relabel “99%” as “68%.” It cannot recover a useful routing signal when high and low scores no longer distinguish right from wrong.
  • Typed JSON guarantees a valid option, not a correct decision. A threshold on an uninformative probability only automates mistakes more confidently.

Important caveats

  • Anthus did not test the Decisions API. OpenAI says that endpoint uses a specialized version of Luna, and no public documentation was available for a direct test.
  • ProofWriter is synthetic, templated logic. It measures chained inference, not every production classification task.
  • Reasoning was disabled to match the fast, one-shot shape of a decision endpoint; enabling it could change both quality and latency.
  • Anthus builds confidence-routing systems and publishes a competing Jev benchmark, so the transparent records matter more than the headline alone.

The deployment lesson

  • Build a labeled test set from the decisions your application actually makes, including the hard tail.
  • Measure accuracy by difficulty rather than relying on an aggregate score dominated by easy cases.
  • Check both calibration and ranking: does higher confidence really correspond to a higher chance of being right?
  • Keep human review or a stronger fallback until the threshold is validated on held-out production-like data.

“A 99% that means 68% will push the wrong decisions past your reviewers.”

The article’s strongest contribution is not “Jev wins.” It is the distinction between a badly scaled confidence score and a score that contains no useful information on hard cases. The first can be recalibrated; the second cannot.