After posting the feed reader Sprinklz to Hacker News, the author of Sammy’s Blog expected a temporary burst of visitors. Instead, the site stayed slow for days, despite only a modest number of active users. Logging individual request paths revealed that much of the traffic wasn’t even fetching articles or running the browser code needed to use the site.

  • Nearly half the traffic came from one unnamed service that repeatedly checked websites for changes to summarize into an AI newsletter, the author reports.
  • Other visitors included identified AI crawlers, search crawlers, and vulnerability scanners probing paths the site didn’t use.
  • The response combined blocking the largest offender, crawler instructions in robots.txt, rate limits, and performance improvements. The post gives no measured before-and-after results.

The useful distinction is between an audience spike and an extraction workload. Counting logged-in readers didn’t explain the pressure; request-level logs did. The author ultimately wants a site that can handle more legitimate traffic, rather than treating every performance problem as something blocking alone will solve.

The 129-comment thread on Hacker News adds experience with the collateral damage of bot defenses—and dissent about whether fighting every request is worth it.

What the thread adds

  • switz — on the downside of blocking whole cloud networks: “It’s frustrating because some of my real users route through these clouds via their work VPNs.” They also report bots using residential addresses, making the distinction harder.
  • kazinator — abandoned compressed-data traps for crawlers: “Serving the data, even though it is compressed and the compression is precomputed, was still a waste of bandwidth. It could hit the browsers of legit user (false positive).”
  • beybol — suggests separating indexing crawlers from user-requested fetches: “OAI-SearchBot/GPTBot/ClaudeBot is just indexing but ChatGPT-User/Claude-User etc. may be valuable traffic generated by user’s conversation with LLM’s.” spindump8930 cautions that they’ve seen different traffic patterns under the same identifier and suspect impersonation; that remains their theory.
  • timpera — the counterexample to escalating defenses: “I have tried a ton of stuff (Cloudflare, Anubis, fail2ban…) to combat the bots lately, but they all have their downsides, and users complained too much, so I just gave up.” Their alternative: “performance improvements on my server and serving every single request.”
  • naet — pushes back on treating the whole problem as new: “All that from one HN comment, very much pre-AI too, not to mention the standard daily slew of requests probing for what I assume are known vulnerabilities in Wordpress or PHP.”

What the article doesn’t disclose

bix6 asks who operated the service responsible for nearly half the traffic. The submitter sammy0910 replies, “haha im not naming names”. Without that identity or a request-rate breakdown, the account is a useful debugging story, not an independently auditable attribution of the offending service.

Handles are pseudonymous; HN publishes no per-comment scores. The fetched order is HN’s ranking, while these excerpts are arranged for usefulness. This is a slice of the thread, not a consensus.