95pctai’s report, attributed to @AI_Andy__, asks whether Jev-like classifiers can decide which technical-document sections answer a question. Its useful move is to measure two different mistakes: hiding an answer and showing irrelevant material.

I read the complete report and inspected repository commit b0c8df2d8c5f3a84e77e4b333d3905e0f4d8b5f4. The repository contains a README and presentation HTML, not the evaluation runner or pair-level outputs. All performance figures below are the author’s reported results; I did not reproduce inference or independently validate the reference labels.

Low misses can hide a lot of noise

The author reports 38 questions, 217 sections from two documents, and 8,246 question–section pairs, including 290 answering pairs. Only an unrelated label hides a section; a direct answer mislabeled tangential still reaches the reader.

At 38 questions per request, the report gives:

  • Jev: four missed answers; 22.8% of unrelated sections retained.
  • Gemma 4 31B baseline: 11 missed answers; 8.1% unrelated retained.
  • Drex 1.5: four missed answers; 69.9% unrelated retained.
  • Cloudflare clef: 48 missed answers; 4.3% unrelated retained.

These are different operating points, not a universal leaderboard. The author’s 25% noise ceiling is a judgment call, not an experimentally established break-even point.

Ranking and default decisions differ

The report finds clef’s default labels too strict, yet its probability ranking close to Jev’s. An exploratory cutoff tuned on pilot sections recovers many answers on the remaining sections. The author explicitly treats this as a hint requiring a planned follow-up—not established equivalence.

Likewise, zero-shot failures of small encoder models do not prove that domain-fine-tuned versions cannot work. Typed output is not a correct decision, and a confidence score does not choose the right product threshold for you.

Speed and cost need a fair comparison

The author reports Jev’s summed request latency at 18.8–26.4 seconds per question and API-accounted model cost at $0.065–$0.077 per full run. That is amortized grading work, not a single request’s response time or total system cost.

The LLM baseline ran on a rate-limited free tier, with some sections entered manually and their time excluded. Its cost was estimated from third-party token prices. The report therefore does not establish that Jev is cheaper, or prove the speed gap would survive equal paid endpoints.

The reference labels are a limitation, too

  • LLMs generated and revised the reference labels; there was no complete human domain-expert pass.
  • Disagreement-driven reviews and a disputed list question complicate interpretation.
  • Corrected noise estimates cover only some systems, so they are not uniformly comparable.
  • Few documents and mostly single runs cannot settle small differences between strong models.

The practical takeaway is a better evaluation design: preserve answers, measure unwanted material separately, calibrate thresholds on separate data, and compare paid endpoints on equal terms. The report is useful evidence for that design—not a reproduced general accuracy claim.