thomasjjj’s human-rights classification project starts with a useful constraint: let models assess observable propositions, combine them in ordinary code, and leave legal determinations to people. Its substantive evidence is a benchmark of local Jev-style alternatives, not a hosted Jev victory: the author explicitly says Jev access is pending and no hosted requests were made.

I inspected the classifier, benchmark protocol, preparation/runner/metric code, full report, and saved predictions at revision c61f56b2f6f37ba00220b057876e555db6384b41. Nine benchmark-validity tests passed locally, and I recalculated selected metrics from the author’s saved predictions. I did not rerun inference, training, latency measurements, or the hosted Jev classifier; this is not a security audit.

Test the task, not the model’s label

The author’s frozen English evaluation uses 100 ETHOS development messages, 300 held-out ETHOS messages, and 145 HateCheck cases. ETHOS contains annotated social-media text; HateCheck is a synthetic suite of specific language behaviors.

  • Every decision-model request asks one direct hate-speech question plus three propositions: identity targeting, attacking language, and speaker endorsement.
  • Gold labels and dataset metadata are excluded from model input; development data alone sets alternate thresholds and temperature scaling.
  • The minimum of the three proposition scores is a gate-like heuristic—not a calibrated joint probability.
  • Hate labels do not validate individual legal elements, genocidal intent, or the full human-rights taxonomy.

A good score can still hide relevant evidence

The author reports d3-mini / CUDA Q4 macro F1 of 0.891 on ETHOS—F1 balanced equally across the two classes. My recomputation from saved predictions confirmed 16 false positives and 16 false negatives.

At the author’s fixed 0.20/0.80 review band, those saved predictions send 77 of 300 messages to review, make 8 errors among 223 automatic decisions, and confidently reject 5 hateful messages. These are verified calculations over supplied outputs, not independently reproduced model behavior.

The author’s English Laya CPU run has zero observed automatic errors among 112 decisions, but leaves 188 messages for review. Zero observed errors in a small selected sample is not a safety guarantee. Its three-question minimum rule also reduces reported hate recall from 0.769 to 0.500: decomposition can lose evidence rather than merely add clarity.

Local and small are different choices

The report also compares TF-IDF word-pattern features, frozen MiniLM embeddings, and SetFit fine-tuning at matched annotation budgets. With 597 labeled training messages, the author reports SetFit mean macro F1 of 0.752 on ETHOS and 0.744 on HateCheck; frozen embeddings score 0.764 and 0.682. Fine-tuning helps transfer here, but not every metric.

These supervised models answer one learned task; decision-model requests answer four questions. The author’s latency and throughput figures are not reproduced here, and are not matched-workload production comparisons. The controlled device tests also report changed decisions for some models, so a provider or runtime swap needs numerical checks as well as timing.

Public English datasets, small functional subsets, uncertain upstream training overlap, and already-visible test results constrain the conclusions. Multilingual investigative corpora and missing conversation context remain unvalidated.

Typed output is not a correct decision. Before filtering an evidence collection, measure which relevant messages disappear, annotate the actual propositions, validate thresholds on the intended population, and keep human review and source evidence intact.