A. Sina Booeshaghi’s author.link ranks preprints against a scientist’s written interests using paper titles and abstracts. The useful boundary is explicit: it recommends what to read, not whether the science is good. I read the full writeup; I did not run the service or reproduce its benchmarks.

What the system does

  • The service checks arXiv, bioRxiv and medRxiv for new papers every four hours, according to Booeshaghi, and offers an optional daily email.
  • The reported comparison uses jev-1.13.0 with noul and gpt-6-luna with predicate, giving both models the same inputs and relevance criteria.
  • A typed relevance score makes sorting straightforward. It does not guarantee the decision is correct—or that a score of 0.8 means an 80% chance of usefulness.

Two tests, different kinds of evidence

  • Personal-interest proxy: Booeshaghi selects 66 papers: 22 he coauthored, 22 keyword-matched controls and 22 controls without the keyword matches. For separating his papers from both control groups, he reports ROC AUC 0.938 for Jev and 0.893 for OpenAI. AUC measures separation across possible cutoffs, not the accuracy of a chosen reading list.
  • Human relevance judgments: On RELISH, which asks whether papers are relevant to other papers, Booeshaghi reports average AUC 0.893 for Jev, 0.872 for OpenAI and 0.721 for shared-word similarity (TF-IDF) when distinguishing relevant from irrelevant papers.
  • Including partially relevant papers among the negatives lowers the reported RELISH AUCs to 0.799, 0.785 and 0.653, respectively. The ambiguous middle matters.

The caveats are part of the result

  • Coauthorship is a proxy for interest, not an independent relevance label. A keyword-matched control, Gravlax, scored highly with both models; Booeshaghi says its author’s work substantially overlaps his own.
  • RELISH supports a related ranking task, not proof that the service finds useful papers for diverse scientists. The writeup does not establish an advantage over modern embedding-based recommenders.
  • Booeshaghi reports OpenAI had faster median responses, while Jev had fewer slow responses and lower cost. Those workload-specific observations were not reproduced here.

The next useful test is readers labeling recommendations and misses on held-out papers. Sorting titles and abstracts is a bounded Jev-shaped task; identifying trustworthy science remains the reader’s job.