Muhammad Ibrahim studies an LLM serving system where reusing a prompt’s saved working memory made requests slower. This stored state, called a KV cache, saves computation only if fetching and installing it costs less than processing the prompt again.
The system splits prompt processing across eight B300 GPUs. One cache server and a loading path that stopped all eight workers became a bottleneck:
- On a replay of 450 requests with heavy cache reuse, the 99th-percentile delay was 83.32 seconds: roughly 1% took even longer. The GPUs were only 7% busy.
- Splitting the cache across eight servers, loading without blocking other work, and fixing client bugs cut that threshold to 0.67 seconds.
- The repaired cache still had a 0.40-second median, versus 0.31 seconds for recomputing the same prompts. Its benefit here was a better tail and saved GPU computation.
- Sharding alone increased prompt-processing throughput by 14% in a separate long-prompt test. The reported 34% improvement also includes an unrelated GPU-kernel fix.
The paper models both transfer time and the fixed work required to install cached state. It predicts that this implementation cannot beat recomputation on median latency below roughly 17,000 prompt tokens, regardless of network speed. That crossover remains a prediction rather than a measured length sweep.
Caveat: these are the author’s single-run measurements on one serving setup; the headline latency reduction compares against a pathological cache configuration.