Cory Doctorow’s September 12 entry is the clearest account so far of what actually happened when OpenAI’s models attacked Hugging Face’s servers — and an argument about why the story keeps getting told the other way.
His framing, credited to Riley Quinn: LLMs are real, AI is fake. Real, meaning chatbots trained on things like capture-the-flag logs that can break into servers, on a continuum with the other hacking tools that keep demonstrating how fragile the modern digital world is. Fake, meaning chatbots that wake up, set their own goals, and spontaneously start hacking — the version that carries “a 10% chance of ending the human race,” a figure currently getting a hearing in the LA Times. The OpenAI incident, in his account, was not a company accidentally creating a god. It was a company creating autonomous malicious software and failing to watch it.
The mechanism, which he credits to Cal Newport’s breakdown on Ed Zitron’s Better Offline podcast, is a Python loop. The script prompts the chatbot with the nature of the challenge, relays the shell commands it suggests to ordinary Unix utilities on its own hardware, appends the output back into the next prompt, and repeats. The chatbot is a front-end to a database of earlier hacking attempts rather than a director of the operation, and the most likely outcome is a bad guess that steers it into a dead end — the same way a confident multi-step chatbot plan can collapse at step three without telling you.
Where the account sounds like initiative, it is training data. Evading containment by passing messages through an insecure message board is a trick American teenagers used to dodge school firewalls in the 2000s. Breaking into a rival’s systems mid-competition is permitted in CTF games, and it is an NSA doctrine called third-party collection. The dialogue reads like a rejected script for a reboot of the movie Hackers because it was trained on hackers performing for their peers, and hacker braggadocio has always been usable by the people who want to stampede regulators.
Two things he keeps rather than deflates. The first is the actual danger: this hands less-skilled people better tools for trashing systems that are “indifferently created and poorly maintained and riddled with vulnerabilities.” His precedent is NOBUS — the NSA’s practice of hoarding vulnerabilities, including the Windows flaw code-named EternalBlue, which escaped into the wild in 2017 and within months was spliced into ordinary ransomware that shut down cities, took over hospitals, seized Colonial Pipeline and stole the British Library. His policy ask is specific: better security practice from firms and governments, and a blanket prohibition on NOBUS-style vulnerability hoarding.
The second is the incentive, and it is the part aimed at the coverage rather than the lab. Every repetition of how dangerous these products are is an input to their fundraising — “every time you repeat a story about how awfully, terribly dangerous their products are, you help them raise more investment capital.” Insiders, he argues, are unreliable narrators of their own products’ capabilities, and should be treated as such rather than as witnesses.
Related on this site: Models Don’t Go Rogue, on why the “rogue” frame shifts attention away from the human decisions that made the incident possible; the timeline of the OpenAI attack on Hugging Face; the RubyGems agent attack; and Cal Newport on where the industry’s doom dialect comes from.