Abi Aryan (“Inference Infra Queen”, Maven inference-engineering instructor, NOVA IMS) teaches LLM latency from first principles on Hamel Husain’s channel — a 41-minute Colab demo that separates prefill from decode and shows why decode owns your latency and cost.

Where the time actually goes

  • Opening puzzle: the same TinyLlama model, same T4 GPU, same inference code produced one request at 3.3s and another at 8.8s — no tricks, no patching
  • Most people’s mental model (“more tokens = more compute”) is wrong; the request shape decides where time goes
  • The inference shape = 4 variables: input tokens, output tokens, batch size (concurrent requests), context length (what attention can see)
  • Real workloads cluster into 4 shapes: chat (~1:1), RAG (~20:1 input:output), creative writing (tiny prompt, long output), agentic (no fixed ratio — the hardest to predict)
  • Agentic shapes are global, not static: the request at step 6 contains every token from steps 1–5

Prefill vs decode: two different bottlenecks

  • Prefill: whole prompt in one parallel forward pass, produces the KV cache; compute-bound and GPU-saturating; scales with input tokens; experienced as time-to-first-token
  • Decode: autoregressive, one token at a time; memory-bandwidth-bound — every token streams the full weights + growing KV cache (2.2GB of HBM traffic for one TinyLlama token); scales with output tokens; experienced as streaming speed
  • “A GPU is just not another faster CPU” — the two phases have different bottlenecks, so different optimization strategies
  • TTFT and streaming speed are reported as separate numbers because they come from different phases

The Colab demo: plain Transformers, no engine

  • Deliberately runs Hugging Face Transformers in eager mode — no vLLM/SGLang — because vLLM hides the prefill/decode split (and doesn’t install cleanly on Colab); seeing the raw loop first is what makes the engines legible
  • TinyLlama 1.1B in FP16 = 2.2GB VRAM on a T4; the inference function syncs to the GPU clock (torch.cuda.synchronize) and times prefill vs decode per request
  • Smoke test first (22 in / 8 out tokens): prefill 152ms ≈ 144 tok/s, decode 7 tok/s — same model, same GPU, different phase, different bottleneck
  • Always warm up the GPU before experiments — skipping warmup in production is a classic footgun

The three shapes, measured

  • Short-in/short-out (chat): prefill ~40ms, decode dominates — prefill ≈ 3% of latency
  • Long-in/short-out (RAG): prefill share grows to ~16% — the only shape where prefill visibly matters
  • Short-in/long-out (creative writing): prefill ≈ 0.5% — and this shape has the longest total latency, because decode is sequential
  • Read vs write asymmetry: ~1,205 input tokens prefill at ~0.44ms/token — writing a token costs roughly 300x more GPU-clock time than reading one
  • That’s why pricing pages price input and output tokens separately

Small requests waste the GPU

  • Prefill throughput: 143 tok/s (25-token prompt) vs 2,271 tok/s (long prompt) — a 16x difference in the same phase
  • A tiny request leaves the GPU nearly idle — the observation that birthed continuous batching (vLLM/SGLang’s scheduler.py batches as many prefill tokens as possible into one request)

KV cache: growth depends on architecture

  • KV cache = memory growing with output tokens; ~0.17MB per token for TinyLlama, measured under theory
  • Growth is architecture-dependent: TinyLlama uses GQA (KV pairs shared across a group of heads); MQA/MLA compress differently; MoE models (e.g. DeepSeek) generate less KV cache
  • When picking a model, prefer architectures that group the KV cache — “think about this as the context filling”

Agentic loops: where cost really lands

  • Simulated 5-step agent loop (build prompt → inference → append to history), no framework — frameworks add latency, hide token consumption, and fail fast in production
  • Every step re-sends the full history from scratch (no context graph, no memory card) — runtime grows ~O(n²) in input tokens
  • Result: prefill ≈ 3% of agentic latency; decode accumulates every step; cumulative latency hugs the decode line
  • Agent shapes add three more variables: batch size, context window, number of tool calls (more tool calls → more latency)
  • Costed on OpenAI pricing: a 5-step loop ran up 11.9x the cost of one chat request — almost all from re-sent input tokens feeding decode
  • The agent-optimization target: the smallest amount of KV cache you can share back to do the decode

The dense takeaways

  • Latency lives in decode; output tokens are ~2 orders of magnitude more expensive than input tokens
  • TTFT is a prefill story; total latency is an output-length story — cap max tokens, don’t just tune the retrieval pipeline
  • Small requests waste the GPU — batch (143 → 2,271 tok/s) or let the engine’s scheduler do it
  • Agents rebuy their entire history every step — context management is survival, not optimization

Homework

  • Try batch sizes 4 and 8 vs 1 and watch prefill/decode latency change
  • Swap engines: llama.cpp on Mac (auto-batching/scheduling), vLLM/SGLang on Linux — how does each affect the phases?
  • Try INT8/INT4 quantized versions of the same FP16 model — memory, latency, cost per phase
  • Explore prefix caching: smallest KV cache you can hand back; read vLLM’s batch.py and the paged/flash attention kernels

Q&A: production mistakes and scaling

  • Mistake #1: shipping models with no smoke test/warmup before pods report ready
  • Mistake #2: serving the same model on different ports without KV-cache sharing across them
  • Open problem: how long to hold KV cache before eviction — dead weight, the same question database companies faced
  • Self-hosting economics: hybrid approach — small one-off requests on API GPUs, long-running multi-step agentic systems on your own GPUs (EU regulation is the deciding factor in Europe)
  • Speculative decoding & chunked prefill: built to control KV-cache growth on very large models (~45B+); not a major concern for small ones
  • Resources: her Maven courses (forward-deployed; inference engineering & systems design), gpengineering.com with her PMPP notes, plus intro and kernel-engineering books

“Context management… that’s what will make the difference between whether you’re building a system that would survive or whether you’re building a system that will eventually get killed.” — Abi Aryan, on agentic systems