Martin Keen explains prompt caching through the work a large language model does before it writes an answer. In this nine-minute IBM Technology lesson, he follows a repeated document from its first processing pass to later questions. The practical advice is simple: put shared material first so later requests can reuse the model’s earlier work.

Cache the input computation (00:13–03:24)

  • Keen starts with a familiar database example. A system can save a query’s result and return that saved result when the same query arrives again.
  • A language-model application can also save completed responses. That is output caching, where another request receives an earlier answer without another generation step.
  • Prompt caching saves computed state for the input. The model still generates an answer, but it can skip work already done on the shared part of the prompt.
  • A transformer computes key-value pairs, stored attention data, for input tokens at each layer. Keen describes these as an internal representation of the prompt’s relationships and context.
  • This happens during prefill, the model’s first pass over a prompt. Saving those key-value pairs allows later requests to reuse that processing.

Reuse a document across different questions (03:24–04:52)

  • A question such as “What’s the capital of France?” contains little input to process. Keen uses it to explain why caching is less compelling for a tiny prompt.
  • His larger example places a 50-page document in the context window, the input space available to the model. The first request asks for a summary.
  • A later request includes the same document but asks a different question. The document remains the shared prefix, the identical beginning of both requests.
  • The system can retrieve the document’s cached key-value pairs and process the new question at the end. Repeated use of substantial input creates the opportunity to save time and cost.

Keep reusable material together (04:59–05:59)

  • Documents are one candidate: manuals, research papers, and contracts can support many questions about the same source.
  • System instructions are another. They define an assistant’s role, rules, and behavior, and may remain identical across many requests.
  • Few-shot examples, sample inputs and desired outputs, can also stay fixed when an application needs a particular response format.
  • Keen includes tool definitions and conversation history among the other prompt components that may be cached. Their usefulness depends on whether the relevant portion stays unchanged.

Put changing questions at the end (06:05–07:52)

  • Automatic caching uses prefix matching. It compares a request with cached input from the beginning, token by token, until the inputs differ.
  • Keen orders his example as system instructions, a manual, formatting examples, and finally the user’s question. A new question can then preserve the earlier shared material.
  • Putting the changing question first breaks the match near the start. The unchanged manual farther down the prompt loses the benefit of that shared-prefix arrangement.
  • Prompt order therefore affects reuse. Keeping stable content ahead of changing content makes the cached portion useful across a sequence of related requests.

Check the provider’s cache rules (07:58–08:57)

  • Keen gives 1,024 tokens as a typical minimum and describes retention ranging from minutes to as long as 24 hours in some systems.
  • Some providers detect reusable prefixes automatically. Others require callers to mark the sections they want cached explicitly in API requests.
  • These limits belong to the implementation. The lesson explains the mechanism without establishing one universal minimum, expiration policy, or savings rate.

Caveat: This is IBM Technology’s conceptual explainer, not a measured performance comparison; cache eligibility, retention, and billing depend on the provider.