Grace Ableidinger explains the request-to-response pipeline in this eight-minute IBM Technology primer.
Turn weights into a service
- Inference runs a trained model on new input; an inference engine schedules work and manages hardware resources.
- vLLM, Ollama and llama.cpp are examples of software used to serve models.
- Tokenization converts text into token IDs; the tokenizer affects prompt length and token-based billing.
- Hosted systems authenticate and route requests before scheduling computation; local setups can combine these stages.
Keep the hardware busy
- Batching groups work from several requests to use a GPU efficiently.
- Continuous batching replaces completed requests with new ones rather than waiting for a whole batch to finish.
- Ableidinger compares single-request serving to running an industrial oven for one pastry.
Separate the two phases
- Prefill processes the prompt and creates a KV cache, the attention state reused during generation.
- Decode generates successive tokens, referring to that cached state and updating it.
- Prefill often needs substantial parallel computation; decode is often limited by moving data through memory.
- The actual bottleneck depends on batch size, model architecture, context length and hardware.
Measure the user’s wait
- Time to first token measures the wait before output begins, including queueing and other request overhead.
- Inter-token latency measures the gaps while the response streams.
- Faster arithmetic alone will not fix a memory-bound decode phase; measure the full serving path.
“Prefill wants raw compute, while decode wants fast memory access.” — Ableidinger, 06:17