Vincent Schmalbach documents something quietly structural: TIME.com now serves two different websites. Humans get the full 303KB HTML page; AI assistant crawlers get a 13KB stripped-down markdown copy — byte-for-byte identical for ClaudeBot, PerplexityBot, and OAI-SearchBot — with ads baked in that no person ever sees. Fetching the same URL from the same machine, changing only the User-Agent header, he shows Googlebot still gets the real page while assistant crawlers get text/markdown served by Mobian, an ad-tech vendor. The headers reveal the economics: a fresh impression UUID on every request, cache-control: no-store, and an x-mobian-tokens count — the unit being billed is tokens fed into a model, not pageviews. Sponsored content that never appears in the human HTML — an Ally Bank FAQ on the Best Inventions collection, a Project Management Institute “Reference Facts” table — sits inside the markdown, and the policy is per-bot: GPTBot and ChatGPT-User are 406-blocked while OAI-SearchBot is waved through. The ads are labeled sponsored; what’s hidden is the audience split. With bot traffic already outnumbering human traffic on most days, Schmalbach argues this is the first clear look at what the web becomes when the main audience is AI models.
TIME Is Serving AI Bots a Different Website, with Ads Built In — Vincent Schmalbach
TIME serves AI crawlers a stripped-down markdown site with token-billed ads that no human ever sees — a first look at the bot-first web.