Cal Paterson’s thesis is simple: AI agents should start with memories, not a blank slate — but almost every agent-memory system on the market gets it wrong. His fix is a file format, not a framework.

He groups today’s memory systems into three failing camps:

  • Harness-locked memory that mines your chat history — mostly remembers things about you instead of the world
  • Complicated pipelines — one prominent system needs a vector database, a graph database, and its own LLM just to decide what’s worth remembering
  • “High Modernist” memory — distilled facts and graphs that strip information from its context until it’s senseless

His alternative, “memoryfields,” is a folder of Markdown pages plus an optional search index that finds pages by meaning (semantic search) rather than keywords. Agents write memories directly in Markdown — their favorite format — instead of feeding text through chunking and summarization machinery.

The design choices:

  • Prose, not chunks. Pages cap at roughly 8KB (about 1,300 words) — short enough to search, long enough to carry real context
  • Semantic jump, not graph walking. Instead of clicking through a knowledge graph one link at a time, semantic search jumps straight to all relevant pages and the agent reads them in parallel — at most two steps
  • More model, less mechanism. Plain files let agents invent their own access patterns (real examples include perl find-and-replaces and querying inline CSV with SQLite), and they get better as models get better
  • Open and portable. A zip of files with a written spec, storable locally, on S3, on GitHub, or over HTTP

The familiar RAG pattern (fetching relevant documents and feeding them to a model) is usually read-only — it makes existing documents legible to an agent. Memoryfields are for writing too: the agent records what it learned, in its own words.

His caveat is worth repeating: never share your context window — including your memories — with parties you don’t trust. There’s still no way for an agent to reliably tell a good prompt from a malicious one.

Memory is data. The less machinery sits between the agent and that data, the better the agent behaves.