Nick Cammarata helped develop the idea that neural networks pack unrelated concepts into the same internal units. He now thinks several famous examples were misread. His essay argues that understanding how language models work may be more tractable than researchers assumed—not because their size disappeared, but because their organization may be easier to follow.

  • A celebrated neuron responding to cats and cars appears, on closer inspection, to recognize a shared V shape. The apparent jumble had a common pattern.
  • Better inspection tools and modern network designs made previously confusing image-model layers much more legible. Cammarata argues that some disorder belonged to the microscope, not the model.
  • Preliminary work on Qwen3-32B suggests roughly 90% of the positive or negative sides of neurons have one main job. He explicitly warns that this estimate depends on definitions and thresholds; it is not a precise measure of how much of the model is understood.
  • His proposed method is experimental: inspect original neurons and their connections, form an explanation, then change or remove components to see whether the predicted behavior follows.

The useful distinction is between related ideas branching outward during learning and unrelated ideas being squeezed together merely to save space. If the first explains much of the apparent mess, researchers may not need to train a second model to untangle the first before studying it.

This is a working hypothesis, not a declaration that AI safety is solved. Cammarata reports less progress on attention—the mechanism that moves information between positions in text—and says the largest language models are nowhere near fully mapped. The interesting contribution is a research direction with testable examples, not the cleanliness percentage by itself.

The 138-comment thread on Hacker News

What the thread adds

  • bluejay2387 — “The claims seem to be an overreach of what is actually proven. … I don’t think these have all been considered sufficiently to start claiming we are around the corner from mechanistic interpretation.”
  • qt31415926 — “Nick isn’t trying to re-claim progressive feature complexity. He’s saying that many polysemantic examples were misread because we assumed neurons packed unrelated things together. … That paper doesn’t analyze neuronal firing patterns whereas Nick’s trying to analyze what fires together and what doesn’t fire.”
  • tuvix — “This is the primary reason, to me, that open weights are so important. If OpenAI and Anthropic want these models to be as autonomous as possible they also need to provide the means to understand them fully.”

Who checks the automated researchers?

Replying to the launch-post excerpt, mentalgear asks: “How can you be sure they can be trusted and that you even understand them correctly and are not biased towards making you feel like they are trustworthy?” The essay describes tests that can disprove neuron explanations, but that broader question about trusting AI-built research tools remains worth separating from whether an individual explanation passes a test.

HN handles are pseudonymous, and HN publishes no per-comment scores. Ordering in the fetched discussion is HN’s own ranking; the selections above are ordered for usefulness. This is a slice of the thread, not a consensus.