Bots

Jason Koebler’s essay in 404 Media is deliberately not about AI doom. While the industry argues over whether there is a ten percent chance models kill everyone, he points at the thing that is already certain: programs that can log into your accounts and act on the open web are annoying as hell, and they are changing what being online feels like. His argument is not that AI will fail. It is that the agentic internet is already here, and it looks less like a helpful assistant than like a thousand automated requests nobody asked for. ...

Nick Abe runs a small Android puzzle app called Dayzle. Two weeks of Google Ads on a CA$40-a-day budget produced 56 billed installs. When he went into the raw analytics, 33 of them did something no person does: opened the app once, spent zero seconds on any screen, and never came back — across 28 phone models in 19 states. The 13 installs that were actual people finished 92 puzzles between them. The 270-comment thread on Hacker News is where the practical detail lives: countermeasures, near-identical experiences at other budget sizes, and one question nobody could answer. ...

Rick Manelius coins the acronym “AI;DR (AI; Didn’t Read)” for a growing frustration: people pasting walls of raw Claude output into Slack, newsletters, and social posts without reviewing or editing it. The post is short but the reaction is visceral — he describes physically flinching when respected colleagues send unfiltered AI copy. The rule is simple: If you’re not bothered enough to review and edit it, the recipient shouldn’t be bothered to read it. Customer support and template copy are fine for unedited AI. Slack discussions, newsletters, and social posts are not. “It’s your name on it; are you proud of the prose and weird AI-isms sprinkled throughout it?” The comments surface a deeper observation (from user Danny): AI creates “borrowed competence” — the gap between what someone can produce and what they actually understand widens. Outputs look sophisticated and use all the right terminology, but the sender may lack the underlying mental model. A vague thought like “we should probably collect the right data” becomes a highly specific 15-point technical requirement — with all the healthy hesitation and uncertainty erased. ...

Nick Gray runs PatronView, a 1.5 million-page database of American philanthropists, and this essay is his year-long war log against everything reading it. In one week his server answered 2.5 million requests while his JavaScript analytics recorded 5,977 pageviews — 214 bot loads for every human one, invisible to Plausible, Fathom, or GA. The centerpiece is the metric he now lives by: pages crawled per visitor referred. Google earns its keep at 46:1 and Bing is defensible at 406:1, but the AI crawlers are in another universe — Anthropic’s Claude-SearchBot read 35,000 pages for every visitor it sent (420,680 pages and 4.63 GB served in one week, against 12 human visitors and 175 KB), and Amazon’s Amzn-SearchBot, feeding Rufus and Alexa answers, was pulling 117,000 pages a day while never sending a single visitor. He blocked both; the polite AI companies respect the 403. Beyond them: a 3.6-million-request day from Chinese botnets, headless Chrome fleets on AWS and Azure, and residential proxy botnets that look exactly like real ISPs. His fixes — block by ASN, challenge every datacenter, CAPTCHA browsers frozen in 2023, and watch the 0.24% solve rate — work, but his conclusion is economic: scraping keeps getting worse because it keeps getting cheaper, so the real fix is pay-per-crawl. Sell Amazon those 3.5 million pages a month at a fair rate. Until then: a crawler that never sends a visitor gets blocked.

Vincent Schmalbach documents something quietly structural: TIME.com now serves two different websites. Humans get the full 303KB HTML page; AI assistant crawlers get a 13KB stripped-down markdown copy — byte-for-byte identical for ClaudeBot, PerplexityBot, and OAI-SearchBot — with ads baked in that no person ever sees. Fetching the same URL from the same machine, changing only the User-Agent header, he shows Googlebot still gets the real page while assistant crawlers get text/markdown served by Mobian, an ad-tech vendor. The headers reveal the economics: a fresh impression UUID on every request, cache-control: no-store, and an x-mobian-tokens count — the unit being billed is tokens fed into a model, not pageviews. Sponsored content that never appears in the human HTML — an Ally Bank FAQ on the Best Inventions collection, a Project Management Institute “Reference Facts” table — sits inside the markdown, and the policy is per-bot: GPTBot and ChatGPT-User are 406-blocked while OAI-SearchBot is waved through. The ads are labeled sponsored; what’s hidden is the audience split. With bot traffic already outnumbering human traffic on most days, Schmalbach argues this is the first clear look at what the web becomes when the main audience is AI models.