Kaoutar El Maghraoui explains why faster chips can leave AI workloads waiting for data. Her twelve-minute IBM Technology whiteboard lesson connects hardware limits to language-model serving.

Measure work per byte (03:01)

  • The memory wall is the gap between a chip’s computing capacity and how quickly memory can supply its data.
  • Arithmetic intensity counts math operations per byte moved from memory. Reusing loaded data raises this ratio.
  • A workload with little work per byte can spend its time waiting for memory, even on a powerful GPU.

Find the bottleneck on a roofline chart (03:44)

  • The chart plots arithmetic intensity against attainable computing performance. Memory bandwidth sets a sloping limit; peak compute sets a flat ceiling.
  • Their intersection, the ridge point, separates memory-limited and compute-limited work in this simplified model.
  • El Maghraoui’s B200 example divides 2.25 petaflops per second by 8 terabytes per second, giving roughly 281 operations per byte.
  • Compare like-for-like specifications. Her example distinguishes a single GPU from an eight-GPU figure and adjusts for sparse versus dense arithmetic.

Separate prompt processing from token generation (08:23)

  • Prefill, the model’s first pass over a prompt, performs large matrix operations. Her example puts it on the compute-limited side.
  • Decode, generating successive tokens, repeatedly reads model weights. At a batch size of one, her example puts it on the memory-limited side.
  • More compute alone helps little when memory cannot deliver data quickly enough.

Change the workload with batching (09:49)

  • Serving multiple requests together lets the model reuse loaded weights across several token generations.
  • That increases work per byte without changing the chip. It can move decoding closer to the ridge point.
  • She names vLLM, SGLang, and TensorRT-LLM as serving systems that use batching and continuous batching.
  • Before benchmarking, estimate the workload’s arithmetic intensity and the hardware’s compute and memory limits. Use that map to choose what to optimize.

Caveat: These are illustrative workload estimates, not measured speedups or a promise that every prefill or decode workload hits the same limit.