OpenRouter’s classification cookbook tests Jev on human-labeled TweetTopic posts. It combines one exclusive category with independent yes/no questions for overlapping tags, then measures which thresholds make those tags usable.

Tune each decision against labeled examples

The provider reports a 150-item calibration sample followed by a separate 400-item batch. These are OpenRouter’s results, not independently reproduced model calls:

  • Music at 0.5: precision was 0.68, meaning many assigned tags were wrong. Recall was 1.00, meaning it caught every labeled music item.
  • Music at 0.8: precision improved to 0.94, while recall fell to 0.88.
  • Celebrity at 0.9: precision was still only 0.41. Raising the threshold could not repair a question that also asked about broad “pop culture.”
  • Exclusive categories: Jev matched 350 of 400 batch labels. That says nothing by itself about the quality of each overlapping tag.

For category review, OpenRouter’s calibration sample had 114 correct answers among 122 items scoring at least 0.8 confidence. That is an observed group result, not a guarantee for the next high-confidence answer.

Make failures visible in the batch runner

The TypeScript example validates responses with Zod, a runtime schema checker. Missing confidence remains unknown; missing usage.cost fails validation instead of silently becoming free inference.

  • Limit concurrent workers and preserve input order.
  • Retry rate limits, server failures, and the specifically identified temporary in-flight-budget error.
  • Stop scheduling new items after a terminal failure, then wait for outstanding requests.
  • Keep unknown confidence in its own review group.

The example groups unknown confidence for inspection, but leaves the final review-routing policy to the application. Its retry helper also accepts numeric Retry-After values only; date-form headers need separate handling.

Budget the model calls, not the whole product

OpenRouter reports $0.010246404 in model charges for the 400-item batch, or about $0.0256 per 1,000 items. It reports 8.6 seconds with eight workers and no retries. The full calibration-plus-batch run reportedly cost $0.0141.

The batch-cost arithmetic checks against the listed input-token rate. It excludes data labeling, application hosting, human review, and future retries. Measure your own text lengths and question set before reusing the budget.

Caveat: provider-reported measurements on a small public dataset; only the displayed arithmetic was checked locally. Typed output is not a correct decision.