Jonathan Ellis of SlopCop compares decision models for Mjolnir, a coding-agent tool that selects GitHub events relevant to an agent’s work. The team can route some events by rule; it uses a classifier for less obvious connections between a session and a teammate’s issue or pull request.
The labels changed the apparent result
Ellis says an early labeling process sent examples to a stronger adjudicator only when the first labeler disagreed with Jev. Most examples where both said no were never checked. That procedure made Jev’s recall, the share of relevant events caught, look better than it did against a separate judging panel.
- Ellis reports Jev recall of 0.91 against the initial labels and 0.57 against the independent panel.
- The later panel used two judges from different model providers, plus a third to break ties after seeing anonymized rationales.
- Judges saw the full session transcript; competing models received the bounded evidence available in production.
These are SlopCop’s results on its own task, not independently reproduced measurements. Model judges can also share errors; the article does not establish human-verified ground truth.
Sample rare positives and inspect the misses
The team sampled using both Jev’s verdict and a separate reranker, then weighted examples back to their traffic proportions. Ellis says relevant pairs made up 11% of traffic. Recording why an event mattered exposed a common weakness: models missed many dependency-related events.
The article reports that SlopCop selected Fastino’s GLiDE for this use case, citing its results and cost, while noting slower service latency than most alternatives tested. The evaluation graphs are specific to this workload; the underlying label set and notebooks are not linked in the article.
A typed relevance answer can still miss an important event. Better labels and better evidence matter as much as the format of the decision.
Verification: read the article and its reported methodology; no model calls or benchmark were rerun.