Nick Gray runs PatronView, a 1.5 million-page database of American philanthropists, and this essay is his year-long war log against everything reading it. In one week his server answered 2.5 million requests while his JavaScript analytics recorded 5,977 pageviews — 214 bot loads for every human one, invisible to Plausible, Fathom, or GA. The centerpiece is the metric he now lives by: pages crawled per visitor referred. Google earns its keep at 46:1 and Bing is defensible at 406:1, but the AI crawlers are in another universe — Anthropic’s Claude-SearchBot read 35,000 pages for every visitor it sent (420,680 pages and 4.63 GB served in one week, against 12 human visitors and 175 KB), and Amazon’s Amzn-SearchBot, feeding Rufus and Alexa answers, was pulling 117,000 pages a day while never sending a single visitor. He blocked both; the polite AI companies respect the 403. Beyond them: a 3.6-million-request day from Chinese botnets, headless Chrome fleets on AWS and Azure, and residential proxy botnets that look exactly like real ISPs. His fixes — block by ASN, challenge every datacenter, CAPTCHA browsers frozen in 2023, and watch the 0.24% solve rate — work, but his conclusion is economic: scraping keeps getting worse because it keeps getting cheaper, so the real fix is pay-per-crawl. Sell Amazon those 3.5 million pages a month at a fair rate. Until then: a crawler that never sends a visitor gets blocked.
99% of My Website Traffic Is Bots — Nick Gray
For every page load Nick Gray sees, 214 more happen unseen — and Claude read 35,000 of his pages for every visitor it sent.