Alex Wang explains the building blocks of language models as memory stores. Each store matches a query against stored keys and retrieves the associated values. Adding more memories creates interference: other entries bleed into the answer you wanted.

His interactive examples build up three mechanisms:

  • A linear layer mixes memories together. Recall deteriorates as more entries compete within its fixed storage.
  • A feedforward network, the layers that transform each token’s representation, can suppress weak matches and reduce that interference.
  • Self-attention, which lets tokens retrieve information from other tokens, selects matches more sharply. Its stored keys and values grow with the text in context.

For 64-dimensional keys, Wang’s simplified experiments put capacity near 64 memories for the basic linear store, about 2,700 for a squared-activation network, and 1.2 million for self-attention. Those figures measure random-vector recall in a single layer, not how many facts a trained language model knows.

The progression makes attention easier to understand: each design changes how aggressively unrelated memories get filtered out. Wang also explains why longer context can require sharper selection, rather than merely room for more text.

The 78-comment thread on Hacker News adds a concrete bridge from the memory analogy to an actual model, plus disagreement about the explanation’s accessibility.

What the thread adds

  • wilkystyle — “In an attention layer, the query, key, and value are all computed at inference time using weight matrices that are determined as a result of the training.” Their dictionary analogy clarifies retrieval: “rather than getting a single value back you get a weighted sum of values.”
  • craftkiller and plasticeagle — two requests for a more concrete starting point. craftkiller asks for “a simplified example of putting a value into a memory store and querying that memory store, showing the actual data structures.” plasticeagle asks: “Similarity to what? The existing keys? Something else?”
  • kooi — the counterargument: “I appreciate this post because starting from linear algebra is the correct way to form an intuition of the higher dimensional semantic spaces we’re dealing with.”

Handles are pseudonymous; HN publishes no per-comment scores, and its ordering is HN’s own ranking. This is a slice of the thread, not a consensus.