IBM’s Kaoutar El Maghraoui explains why moving an LLM’s stored attention state can shape generation performance across different chips.

The memory problem

  • A KV cache stores keys and values from earlier tokens so the model can reuse them during attention instead of recreating them each step.
  • The cache grows with context. Large device memory holds more state; smaller on-chip memory keeps the current working data close to compute.
  • Efficiency depends on moving the required data between those levels, as well as on raw compute capacity.

Three approaches

  • GPU: the serving runtime allocates KV storage; attention kernels and hardware caches influence access. Paged allocation can use fixed-size blocks instead of reserving one large slab per request.
  • TPU: the compiler and kernels shape tiling and pipelining between high-bandwidth memory and fast vector memory, overlapping transfers with computation.
  • IBM Spyre: the compiler schedules explicit loads and stores into per-core SRAM scratchpads; El Maghraoui describes 128 GB of LPDDR5 per card and 32 active cores with 2 MB scratchpads.

What the comparison shows

  • The same conceptual cache can travel through different memory hierarchies and be orchestrated at different layers of the stack.
  • This is an architectural explanation from an IBM speaker, not a measured head-to-head benchmark or a claim that one chip always wins.

“Do not ask only how much math they can do. Ask how they move the model’s memory states.”