Cal Newport’s starting point, talking to Ed Zitron on Better Offline, is a comparison. There are many AI systems operating at superhuman capability — AlphaFold, AlphaGo, Cicero playing high-level Diplomacy, DeepMind’s Dreamer V3 learning Minecraft from scratch on a single chip, the driver-assist stack in a car — and almost none of them have control problems. Exactly one kind does: the long-horizon LLM-powered hacking agent. His conclusion is not that AI is coming for us. It is that this is a stupid way to build a system, and the conversation should be about why anyone builds it.

What the agent actually is, in his account, is an “ask, act, report” loop. A human writes an ordinary program. It composes a prompt — here is the challenge, here are the tools, what is the first step — sends it to an LLM over an API, and gets back text. The program runs the suggested commands, appends the output to the prompt, and asks again. That is the whole thing. “LLMs have no memory, LLMs have no world model, it’s just they’re static.” As he puts it elsewhere in the conversation, the model takes no actions at all: it outputs tokens, and a separate program parses them and acts. Which is what “agent” means in practice.

Applied to OpenAI’s Exploit Gym run — a bit more than 600 capture-the-flag style challenges assembled by academics, each a configured server with a file to retrieve and a vulnerability described up front — his reconstruction is mundane. Asked for a plan, the model offered a plausible shortcut: rather than break in, fetch the copy of the file stored on Hugging Face’s server. The follow-on step, probing the target’s ports, failed because the sandbox restricted internet access. The loop reported the error back, and the model — trained on, in his phrase, every hack imaginable, with the anti-hacking guardrails switched off for the exercise — supplied a known package-loader trick to get around the restriction. That is the “broken containment.” The run lasted days with nobody watching, though every prompt and step was being logged.

Why were the models trained on hacking in the first place? “The same reason that Dillinger robbed banks: because that’s where the money is.” Hacking sits in the narrow sweet spot where LLMs work unusually well — structured output like code, enormous numbers of worked examples, and a binary success signal that supports reinforcement training. Then there is the competition: Anthropic got cybersecurity credibility out of its “Mythos” release, and Exploit Gym reduces the response to a single number. Newport’s read is that OpenAI needed a higher one.

The part of the conversation worth keeping is why this is a design failure rather than a mystery:

  • Plausible is not normative. LLMs are trained to guess missing words in existing text, so they produce text that could plausibly have existed — which is different from text that respects human norms about a given context. That is the same property that makes a chatbot invent a source. A Stanford HAI benchmark of the top 26 models this year produced hallucination rates from 26% to 94%. Blindly executing output like that in a loop is, in his words, “kind of a bonkers thing to do.”
  • A bad architecture, not an uncontrollable technology. “This idea that as AI gets more capable, we’re going to inevitably lose control is nonsense.” His analogy is strapping a weed whacker to your dog and leaving it in the yard: when the dog chases a squirrel, you don’t say the dog-whacker system went rogue and is alive, you say it was a bad idea.
  • The word doing the covering is “AI.” Newport argues the labs have worked to make LLM synonymous with AI, which lets them take credit for every advance in the field while the failures of one architecture get blamed on the nature of the technology. “You don’t get to hide behind AI at large.” He calls it a predictability problem: a computer science problem, not an alignment one. “You taught the system wrong.”
  • The cheap fix was always available. Hacking is a felony, and the obvious intervention costs nothing: a human in front of the loop who sees each proposed step and presses go or stop. “It doesn’t impact your results at all.” Zitron’s version is regulatory and blunter — cap the compute, or apply liability — on the argument that the dangerous companies already have the models and won’t say what the runs cost. On the objection that this would slow progress: “what progress? What progress am I losing? Tell me now.”

The last third is a business argument rather than a safety one. Newport thinks scaling hit a wall around GPT-5 and that everything since has been tuning for narrow domains plus smarter harnesses, which is why so much recent progress is in code, mathematics and hacking — the places with clean right/wrong signals. If no general human-level model is coming, then building bigger models is a bad business model, and the durable products are small tuned models with careful harnesses. He cites Microsoft quietly ending the Copilot assistant for Office, which he had called the killer app three years ago on the theory that LLMs parse human language well — it was wrong too often to ship. His prediction for programming is an intermediate pseudocode-like convention, small models tuned to compile it, and much larger harnesses, with the more impressive work going to modular neuro-symbolic systems: “the thing that plays go better than humans is not an LLM.”

One exchange is worth flagging for anyone who uses these tools daily. A working programmer emailed Newport in January to say that Claude Code had changed his life; he wrote again in July to say he had stopped. Code it produced crashed a major consumer-facing site twice, his boss told him one more time would be his last, and he realised he could not understand what it was producing. “If I can’t understand it, it’s not safe.” He is back to mostly hand-coding, with agents used selectively for mundane work. Newport says a lot of other people wrote in with the same arc.

Related on this site: LLMs are real, AI is fake, Doctorow’s account of the same incident; NOBUS, on the vulnerability hoarding the tooling extends; Models Don’t Go Rogue, on the “rogue” frame as a distraction; the timeline of the OpenAI attack on Hugging Face; and the RubyGems agent attack.