Mack Grenfell’s Trakkr study follows what happens after an AI assistant cites a statistic. The cited webpage often contains the number, but the trail back to the original research breaks or changes its meaning.

  • Researchers asked five assistants 200 questions each, separating 160 everyday questions from 40 suspected examples of statistical folklore. They traced 1,285 cited statistics from the everyday set, up to five pages deep.
  • An estimated 40% reached original research supporting the same claim, with a 95% uncertainty range of 34%–46%. Another 8% reached research whose number or context differed.
  • Forty percent were uncheckable, including inaccessible originals, and 11% led nowhere. Those categories measure different failures; the unmatched remainder cannot be called fabricated.
  • The automated judge initially found only 23% sourced. Correcting its judgments with 150 randomly sampled human checks raised the estimate to 40%.

A citation can preserve digits while losing the population or definition. In one traced example, comments from 15 women rejecting health websites became a sweeping claim about website first impressions. Checking the original study’s wording matters more than finding another assistant that repeats the number.

Caveat: Trakkr sells AI citation-monitoring software, and these are its one-day API results, with human-corrected estimates rather than independent replication.

The 67-comment thread on Hacker News adds practical checks and a missing comparison.

What the thread adds

  • bprathamesh2003 — reports figures appearing in quote-extraction output despite being absent from the document: “we still ask it to cite the exact line it got the figure from.”
  • bradknowles — describes a writing skill that requires checking references and disclosing missing originals: “Primary sources are much preferred over secondary sources, and where no primary source can be found, that must be noted.”
  • ticulatedspline — asks for a human comparison: “I wish they did a control. Gave the question to x number of people and had them provide an answer.” The study leaves that unanswered.
  • furyofantares — challenges the evaluator as well as the assistants: “it also might just not understand the mildest nuance in its context or apply any common sense.” Their example concerns an ice-cream percentage copied from a source but misinterpreted.

HN handles are pseudonymous and no per-comment scores are public. HN’s ranking determines the sampled order; this is a slice of the discussion, not a consensus.