Martin Keen on IBM Technology lays out four retrieval options for AI applications, from putting documents directly in the prompt to running a dedicated vector database.
Start with the smallest workable system
- No retrieval: put a small document collection directly into the model’s context window.
- Keyword search: let an agent search, read results, and refine its query with tools such as grep.
- An existing database’s vector index: add meaning-based retrieval without operating another database.
- A dedicated vector database: consider separate infrastructure when vector volume and query demands justify it.
What makes each step necessary
- Long prompts cost more to process, and irrelevant context can reduce answer quality.
- Literal matching can miss the same problem expressed in different words.
- Keen illustrates that gap with “how do I reset my password?” versus a ticket titled “credentials loop after update.”
- Embeddings, numerical representations of content, can bridge that wording gap through similarity search.
- Documents and queries both need embeddings; changing the embedding model can require rebuilding the document embeddings.
- Index freshness and the memory used by large indexes belong in the operating-cost calculation.
Separate the capability from the infrastructure
- Ask whether the application needs vector search, then whether its existing database can provide it.
- Keen places dedicated systems at the high-volume, frequent-query end of the pyramid.
- His millions-to-billions discussion is a sizing illustration; the talk supplies no benchmark or universal migration threshold.
- The password example illustrates semantic matching, without testing a particular embedding model or a query-rewriting agent.
“The rule is we’re only going to move up the pyramid when the workload actually warrants doing that.” — Martin Keen, 1:31