Social Monitor collects posts across sources and uses decision models to assess reader value. Its five-day Jev comparison shows both a practical ranking pipeline and the traps in comparing it with an older feed.
Score before selecting
- The project’s scorer sends an interest, source text and structured questions to an OpenRouter System One endpoint. It checks the returned model, provider and answer shape before accepting a result.
- The reported review ranked by usefulness and relevance, then applied story deduplication and source caps. A separate model judged a blinded set; it was not a human ground-truth panel.
- The authors report 37 useful unique stories in 40 Jev selection slots versus seven in 40 legacy slots. The legacy feed filled only 16 slots, so the comparison mixes selection quality with underfilling.
Limits on the result
- The older selections were persisted publications, not a fresh replay against the same snapshot. The Jev review also used a different rubric from the implementation and did not test final summaries.
- The authors report that 3,888 scoring calls cost $0.228703, with median latency of 356 ms. Those figures cover the scorer only, not an end-to-end feed.
- A separate 30-item spot check used a model judge and only already-assessed posts. It cannot establish full-feed recall or human usefulness.
Caveat: The repository and reports were inspected, not run. The reported results were not independently reproduced. Valid typed choices do not establish correct ranking decisions.