A faster graphics card helps little if the slowest operation cannot use its resources. Mattias Dürrmeier and Alberto Lerner examine that problem in single-user model generation with long prompts.
They profile one decoder block, the repeated computational unit inside a language model. This lets them inspect model shapes that would otherwise exceed a card’s memory. Qwen3 models with four, eight and 32 billion parameters run across Turing, Ada and Blackwell workstation GPUs; separate GPT-family measurements track architectural changes.
The authors report:
- At the longest tested contexts, Blackwell improves attention speed over Ada by about 12% for the smaller Qwen models and 32% for the larger one.
- The smaller model’s attention uses only 21% of Blackwell’s available memory bandwidth at long contexts. More memory bandwidth on the specification sheet does not automatically become faster generation.
- The fixed-cost parts of a block accelerate substantially on newer hardware. Attention grows with the prompt and eventually dominates runtime, leaving those other gains less influential.
Their recommendations are to split attention work along the sequence, schedule cache transfers in unused bandwidth windows, fuse more operations and reduce synchronization stalls. The KV cache is a model’s stored working memory for a prompt.
These are profiling results and proposed systems improvements. The paper does not demonstrate a complete implementation combining those recommendations. It also notes a trade-off: faster attention would shorten the window available for hiding cache transfers.
Caveat: the attention experiments extend to one million tokens beyond the models’ supported context lengths. An isolated block measurement does not establish that a complete model fits on one card or answers correctly at that length.