The argument in Cory Doctorow’s September 12 Pluralistic entry that stands on its own — separate from the question of whether a chatbot can “go rogue” — is about what happens to the vulnerabilities these tools are being pointed at.

His precedent is a doctrine called NOBUS, short for “No One But Us.” The NSA and the CIA research bugs in widely used software. Sometimes they tell the vendor, so it gets patched. And sometimes they find a good one and keep it secret so they can use it against adversaries, on the reasoning that nobody else is smart enough to find the same flaw, so it can be left unpatched without putting anyone in danger.

How that goes when it fails is documented. In 2017 the NSA lost track of a Windows vulnerability it had hoarded, code-named EternalBlue. After it reached the wild, hackers spliced it into ordinary, unremarkable ransomware, and within months it was inside attacks that shut down cities, took over hospitals, seized the Colonial Pipeline and stole the British Library. Doctorow’s version of the sequence: “It’s as though they found some guy under a Prius removing the catalytic converter with a Sawzall and handed him a piece of software that could shut down major American cities.”

His claim is not that the chatbot labs invented this, but that they are extending it. Autonomous malicious software is already considered irresponsible to build; a chatbot wired to a Python loop that runs its suggested commands is a way of producing it without the skill, pointed at systems that are “indifferently created and poorly maintained and riddled with vulnerabilities.” The tools get better for people who would otherwise be incompetent, and the targets do not get better at all.

Which is why the ask he lands on is about infrastructure and policy, not about model alignment:

  • Better security practice from firms and governments — the fragility, not the awakening, is the actual exposure.
  • A blanket prohibition on NOBUS-style vulnerability hoarding, so that a flaw found by a spy agency gets reported and patched rather than kept as a weapon.

Read against the rest of this month’s coverage, the split is clean. Whether the OpenAI models’ attack on Hugging Face was “autonomy” is contested and largely a question about training data and supervision. Whether states should be sitting on exploitable flaws in the software everyone runs is not contested, and the answer there has been known since 2017.

Related on this site: LLMs are real, AI is fake, Doctorow’s account of why the Hugging Face incident was a Python loop rather than a model waking up; Models Don’t Go Rogue, on the “rogue” frame as a distraction from the decisions that made the incident possible; the timeline of the OpenAI attack on Hugging Face; and the RubyGems agent attack.