Fabio Akita compares decision models on explicit business rules and paired English/Brazilian Portuguese tasks. The useful artifact is not just a ranking: his public repository includes frozen cases, saved responses, scoring code, error examples, and deployment notes.

I read the full article and inspected the request builder and scorers. I also checked archived response/request hashes and recounted selected saved results offline. No model inference, training, deployment, or invoice verification was repeated. The experiments remain Akita’s, not an independent benchmark run.

A small test with useful controls

  • Akita’s original policy suite contains 130 scenario families, each tested with reordered options, opaque labels, and a misleading reviewer comment, plus its original presentation.
  • His bilingual suite contains 84 families, each in two languages and three conditions: compact, worked examples, and distracting context.
  • These are AI-assisted synthetic cases without independent human annotation; variants and translations are correlated, not fresh independent evidence.
  • The compact comparison uses 168 answers; small task cells make sweeping model rankings unreliable.

Close scores, different operating constraints

In Akita’s reported compact results, Perplexity Decider 27B and Decision 3.0 each score 167/168, while Jev and Microsoft Decision-1 each score 164/168. His uncertainty analysis does not establish that the small lead over Jev generalizes.

  • My offline recount confirms 164/168 for saved Jev and Microsoft responses, and 167/168 for saved Decision 3.0 responses; Perplexity’s result is source-reported here.
  • Akita reports roughly 49 GiB of weights for the large local models; his observed client medians are about 1.4–1.5 seconds, versus 255 ms for Jev.
  • Those timings compare different deployments, networks, and serving stacks—not architecture in isolation.
  • Equal aggregate accuracy does not mean interchangeable errors; Jev and Microsoft fail different cases.

A confidence threshold is not a safety certificate

Akita’s small Decision 3.0 Edge checkpoint is fast but makes 14 errors among 68 compact predictions with option probability at least 0.9. My saved-response recount confirms those counts; it does not validate the model on new inputs.

His reported Microsoft results also lose their error-free high-probability pattern when Portuguese examples or distractors are added. Classification accuracy, calibrated answer probabilities, and estimating a fair coin’s outcome are separate tests.

The article calls parts of this “confidence,” but the scorer uses the selected option’s probability, not the vendor’s separate confidence field. Thresholds must name the quantity they apply to.

Token cost is not workflow cost

Akita values saved compact usage at dated list rates: approximately US$0.19 per 10,000 Jev calls versus US$0.065 for Microsoft. My offline arithmetic agrees; the difference comes from reported token counts despite equal listed per-token rates.

  • These are workload extrapolations from saved usage, not actual batches of that size or reconciled bills.
  • Local electricity, hardware, maintenance, and setup cost remain unmeasured—not zero.
  • A misrouted critical ticket can matter more than the API-price difference.
  • Keep permissions and numeric rules in ordinary code; use a model to interpret free text only where needed.

Typed output is not a correct decision. Borrow the evaluation discipline—inspect errors, preserve raw results, test language and context effects—rather than treating a near-perfect synthetic score as permission to automate.