Bots

Nick Gray runs PatronView, a 1.5 million-page database of American philanthropists, and this essay is his year-long war log against everything reading it. In one week his server answered 2.5 million requests while his JavaScript analytics recorded 5,977 pageviews — 214 bot loads for every human one, invisible to Plausible, Fathom, or GA. The centerpiece is the metric he now lives by: pages crawled per visitor referred. Google earns its keep at 46:1 and Bing is defensible at 406:1, but the AI crawlers are in another universe — Anthropic’s Claude-SearchBot read 35,000 pages for every visitor it sent (420,680 pages and 4.63 GB served in one week, against 12 human visitors and 175 KB), and Amazon’s Amzn-SearchBot, feeding Rufus and Alexa answers, was pulling 117,000 pages a day while never sending a single visitor. He blocked both; the polite AI companies respect the 403. Beyond them: a 3.6-million-request day from Chinese botnets, headless Chrome fleets on AWS and Azure, and residential proxy botnets that look exactly like real ISPs. His fixes — block by ASN, challenge every datacenter, CAPTCHA browsers frozen in 2023, and watch the 0.24% solve rate — work, but his conclusion is economic: scraping keeps getting worse because it keeps getting cheaper, so the real fix is pay-per-crawl. Sell Amazon those 3.5 million pages a month at a fair rate. Until then: a crawler that never sends a visitor gets blocked. ...

Vincent Schmalbach documents something quietly structural: TIME.com now serves two different websites. Humans get the full 303KB HTML page; AI assistant crawlers get a 13KB stripped-down markdown copy — byte-for-byte identical for ClaudeBot, PerplexityBot, and OAI-SearchBot — with ads baked in that no person ever sees. Fetching the same URL from the same machine, changing only the User-Agent header, he shows Googlebot still gets the real page while assistant crawlers get text/markdown served by Mobian, an ad-tech vendor. The headers reveal the economics: a fresh impression UUID on every request, cache-control: no-store, and an x-mobian-tokens count — the unit being billed is tokens fed into a model, not pageviews. Sponsored content that never appears in the human HTML — an Ally Bank FAQ on the Best Inventions collection, a Project Management Institute “Reference Facts” table — sits inside the markdown, and the policy is per-bot: GPTBot and ChatGPT-User are 406-blocked while OAI-SearchBot is waved through. The ads are labeled sponsored; what’s hidden is the audience split. With bot traffic already outnumbering human traffic on most days, Schmalbach argues this is the first clear look at what the web becomes when the main audience is AI models. ...