Parallel compares decision models on work it actually does: ranking search results, detecting meaningful page changes, filtering poor content, and routing queries to the right index. Tim Neumann, Vlad Shulman, and Ethan Kam use human-labeled examples from their production pipelines rather than another general-purpose leaderboard.

I read the full evaluation and serving appendix. All measurements below are Parallel’s reports, not independently reproduced here. Its own search business also benefits from the case for specialized models.

A strong default is not the best model for every task

  • On reranking, Parallel reports 71.45 top-K overlap for Jev, versus 70.68 for Perplexity and 70.51 for Liquid d1, using tuned prompts.
  • Top-K overlap measures how much a model’s leading result set matches the human-labeled leading set—not answer accuracy or probability calibration.
  • Parallel describes those decision models as effectively tied; its own fine-tuned reranker scores 75.21.
  • On meaningful page changes with starting prompts, Parallel reports 93.3 AUPRC for Perplexity, 87.9 for Jev, and 56.7 for Kev-4B.

AUPRC summarizes the tradeoff between finding genuine changes and raising false alarms across decision thresholds. It does not establish that a returned “90%” is correct that often.

Improve the question before changing the model

Parallel’s prompt-tuning loop finds errors on a labeled calibration set, asks Claude to turn those error patterns into instructions, and keeps only improvements. The winning prompt is then evaluated once on a separate held-out set.

  • Parallel reports using five rounds and 800 calibration examples for page changes.
  • Rune’s held-out page-change AUPRC rises from 74.1 to 83.9 in its results.
  • Tuned OpenAI Decisions reaches 88.8, above untuned Jev’s 87.9—a useful illustration of prompt sensitivity, not a tuned-versus-tuned win.

The reusable lesson is the separation between examples used to improve the instructions and examples reserved to test the result.

Serving details are part of the comparison

Parallel favors Jev’s hosted combination of quality, cost, and latency. But its hosted latency ranking is a 30-minute OpenRouter observation on October 14, not a controlled infrastructure comparison or a service guarantee.

  • Open-model setups differ in GPU, serving engine, numerical precision, and input limits; the appendix makes those differences inspectable.
  • GLiNER receives a shorter input allowance with middle trimming, which complicates a pure model-quality comparison.
  • Parallel’s claim that its custom reranker costs about a quarter of Jev’s per-token price is not a total-cost accounting: labeling, training, deployment, and idle capacity still matter.
  • The article does not release the labeled examples or a runnable evaluation harness, so the reported margins cannot be reconstructed from the page alone.

Use the study as a model-selection procedure, not a universal ranking: start with a hosted baseline, label real cases, tune without touching the final test set, and consider a smaller specialized model when the workload justifies operating it. Typed output is not a correct decision, and ranking performance is not evidence of calibrated confidence.