Hamel-Husain

Abi Aryan (“Inference Infra Queen”, Maven inference-engineering instructor, NOVA IMS) teaches LLM latency from first principles on Hamel Husain’s channel — a 41-minute Colab demo that separates prefill from decode and shows why decode owns your latency and cost. Where the time actually goes Opening puzzle: the same TinyLlama model, same T4 GPU, same inference code produced one request at 3.3s and another at 8.8s — no tricks, no patching Most people’s mental model (“more tokens = more compute”) is wrong; the request shape decides where time goes The inference shape = 4 variables: input tokens, output tokens, batch size (concurrent requests), context length (what attention can see) Real workloads cluster into 4 shapes: chat (~1:1), RAG (~20:1 input:output), creative writing (tiny prompt, long output), agentic (no fixed ratio — the hardest to predict) Agentic shapes are global, not static: the request at step 6 contains every token from steps 1–5 Prefill vs decode: two different bottlenecks Prefill: whole prompt in one parallel forward pass, produces the KV cache; compute-bound and GPU-saturating; scales with input tokens; experienced as time-to-first-token Decode: autoregressive, one token at a time; memory-bandwidth-bound — every token streams the full weights + growing KV cache (2.2GB of HBM traffic for one TinyLlama token); scales with output tokens; experienced as streaming speed “A GPU is just not another faster CPU” — the two phases have different bottlenecks, so different optimization strategies TTFT and streaming speed are reported as separate numbers because they come from different phases The Colab demo: plain Transformers, no engine Deliberately runs Hugging Face Transformers in eager mode — no vLLM/SGLang — because vLLM hides the prefill/decode split (and doesn’t install cleanly on Colab); seeing the raw loop first is what makes the engines legible TinyLlama 1.1B in FP16 = 2.2GB VRAM on a T4; the inference function syncs to the GPU clock (torch.cuda.synchronize) and times prefill vs decode per request Smoke test first (22 in / 8 out tokens): prefill 152ms ≈ 144 tok/s, decode 7 tok/s — same model, same GPU, different phase, different bottleneck Always warm up the GPU before experiments — skipping warmup in production is a classic footgun The three shapes, measured Short-in/short-out (chat): prefill ~40ms, decode dominates — prefill ≈ 3% of latency Long-in/short-out (RAG): prefill share grows to ~16% — the only shape where prefill visibly matters Short-in/long-out (creative writing): prefill ≈ 0.5% — and this shape has the longest total latency, because decode is sequential Read vs write asymmetry: ~1,205 input tokens prefill at ~0.44ms/token — writing a token costs roughly 300x more GPU-clock time than reading one That’s why pricing pages price input and output tokens separately Small requests waste the GPU Prefill throughput: 143 tok/s (25-token prompt) vs 2,271 tok/s (long prompt) — a 16x difference in the same phase A tiny request leaves the GPU nearly idle — the observation that birthed continuous batching (vLLM/SGLang’s scheduler.py batches as many prefill tokens as possible into one request) KV cache: growth depends on architecture KV cache = memory growing with output tokens; ~0.17MB per token for TinyLlama, measured under theory Growth is architecture-dependent: TinyLlama uses GQA (KV pairs shared across a group of heads); MQA/MLA compress differently; MoE models (e.g. DeepSeek) generate less KV cache When picking a model, prefer architectures that group the KV cache — “think about this as the context filling” Agentic loops: where cost really lands Simulated 5-step agent loop (build prompt → inference → append to history), no framework — frameworks add latency, hide token consumption, and fail fast in production Every step re-sends the full history from scratch (no context graph, no memory card) — runtime grows ~O(n²) in input tokens Result: prefill ≈ 3% of agentic latency; decode accumulates every step; cumulative latency hugs the decode line Agent shapes add three more variables: batch size, context window, number of tool calls (more tool calls → more latency) Costed on OpenAI pricing: a 5-step loop ran up 11.9x the cost of one chat request — almost all from re-sent input tokens feeding decode The agent-optimization target: the smallest amount of KV cache you can share back to do the decode The dense takeaways Latency lives in decode; output tokens are ~2 orders of magnitude more expensive than input tokens TTFT is a prefill story; total latency is an output-length story — cap max tokens, don’t just tune the retrieval pipeline Small requests waste the GPU — batch (143 → 2,271 tok/s) or let the engine’s scheduler do it Agents rebuy their entire history every step — context management is survival, not optimization Homework Try batch sizes 4 and 8 vs 1 and watch prefill/decode latency change Swap engines: llama.cpp on Mac (auto-batching/scheduling), vLLM/SGLang on Linux — how does each affect the phases? Try INT8/INT4 quantized versions of the same FP16 model — memory, latency, cost per phase Explore prefix caching: smallest KV cache you can hand back; read vLLM’s batch.py and the paged/flash attention kernels Q&A: production mistakes and scaling Mistake #1: shipping models with no smoke test/warmup before pods report ready Mistake #2: serving the same model on different ports without KV-cache sharing across them Open problem: how long to hold KV cache before eviction — dead weight, the same question database companies faced Self-hosting economics: hybrid approach — small one-off requests on API GPUs, long-running multi-step agentic systems on your own GPUs (EU regulation is the deciding factor in Europe) Speculative decoding & chunked prefill: built to control KV-cache growth on very large models (~45B+); not a major concern for small ones Resources: her Maven courses (forward-deployed; inference engineering & systems design), gpengineering.com with her PMPP notes, plus intro and kernel-engineering books “Context management… that’s what will make the difference between whether you’re building a system that would survive or whether you’re building a system that will eventually get killed.” — Abi Aryan, on agentic systems ...

Hamel Husain makes the case that “are you building the right thing?” matters more than evals: if users can’t verify an AI answer, your product creates work instead of saving it. Three worked examples and four design principles. ~17 minutes on his own channel. The core problem: verification is the bottleneck Classic AI product: user asks “what was revenue for product A last quarter?”, AI queries a database and returns “$4.21M” — but the user can’t validate the number without redoing all the work Lenny’s tweet: data science teams now spend their time reviewing half-assed AI analysis from PMs and data engineers — 50% of the time it’s wrong Creating output is now easy; verification is the bottleneck — design products with that in mind If a product seems impossible to eval, that’s a product smell: your users have to check it too Example 1: business Q&A → evidence-backed analysis Instead of a bare answer, show supporting information: base the analysis on already-vetted analysis (notebooks others signed off on) State the assumptions — confirm the metric definition (from a semantic layer, linkable to the governed definition) Show intermediate calculations (returns, customer counts) and raise issues the user might care about “Open as notebook” UI idea: an AI-generated notebook (Jupyter/Marimo style) with narrative + queries — step through it, edit it, ask AI questions about it, and have the AI state what it couldn’t verify Put yourself in the expert’s shoes: a data scientist checks prior analyses, definitions, and intermediate calcs — give users those same affordances Real product doing this today: Hex — a chat interface that flows into a notebook; progressive disclosure is an established pattern we forgot in AI apps Example 2: PE lesson plan assistant (K-12) Vanilla version: input grade, class length, location, equipment → AI spits out a lesson plan — no way to judge quality Better: anchor the output in existing curriculum — what are trusted colleagues doing? “Here’s a lesson plan exactly like yours, used at 14 schools, run 30 times, by teachers you trust” Show the edits made for your inputs (“shortened to a 45-minute class, matched to your equipment”) — accept or reject each Cold-start problem: seed with expert-vetted plans up front; it makes the app easier to eval and users more confident Example 3: medical claims report generator The task: read a patient’s chart + claim (thousands of documents) and produce a 52-page report supporting or denying a claim — a doctor can’t eyeball 52 pages Redesign as a research assistant: surface atomic elements first — contradictions (open both pages, resolve), key facts (validate, include, dismiss), open questions (add notes) Do the work the user would normally do and guide them through it; understanding built along the way is arguably more valuable than the final report Four design principles Provenance — show where information comes from (prior analyses, source documents, other lesson plans); the more curated/trustworthy, the better Progressive disclosure — show details at the right time; adapt as your product and users mature Signals and heuristics — mirror how experts sanity-check: other reports, smell tests, social proof, supporting evidence Modularity / small steps — break work into small verifiable pieces, show your work along the way, and sometimes force the human in the loop (e.g. doctors step through findings before the final report) “The goal is to make the human understand, not necessarily to generate a report.” ...

Hamel Husain demos the Codex desktop app’s remote-control mode — the best setup he’s found for untethering AI coding from your computer and running sessions from your phone. ~4 minutes on his own channel. Why remote control When coding with AI, don’t be tethered to your laptop — walk around, close the screen, check things on your phone The Codex desktop app is the best client he’s found for remote AI coding Setup: connect your computers Settings → Connections — he’s connected to a Mac mini and a MacBook Pro (both servers for agentic coding) To add: go to the server, click the link to get a code, then Add it in the app Keep the connection dots green — refresh occasionally; rarely you may need to delete and re-add a connection Keep the Codex app updated (Codex menu → check for updates) Starting a remote session New chat (Cmd+N) or Add new project → Remote, then pick the connected computer and give the folder path You see all sessions running remotely — which machine, which folder, live green dot Live demo Prompt: “Look at my Gmail and find the last 10 emails received from this email address, which are newsletter posts. Put it in the table” — uses the Gmail skill/connector and computer use on the MacBook Pro From the phone (ChatGPT app → Codex tab): see every connected computer and live session; send follow-ups like “Please critique my writing in all 10 emails with subagents. Use the writing skills in this repo to do so” Desktop shows the run: 5.5 High extra fast model, summary view listing the 4 spawned subagents processing email batches; phone shows 2 agents running / 2 finished You can even attach photos or screenshots from your phone “It’s the best remote control I’ve ever used. I haven’t found any limitations.” ...

Shreya Shankar (Stanford CS professor, co-creator of the AI evals course with Hamel) kicks off the 12-part AI product engineering series. 27 minutes on Hamel Husain’s channel. Why this matters Output quality is the biggest barrier to productionizing agents (LangSmith annual report) — and figuring out how to evaluate models is genuinely hard Vendors (LangChain, Braintrust, Arize) are selling end-to-end automated eval tools: point an LLM at your traces, it finds and fixes your bugs The catch is epistemic: what “good” means lives in your head, not in the traces — if a tool could fully fix your product, it could fix everyone’s, and there’d be nothing left to differentiate yours AI’s real job: help you express and apply your judgment faster, not replace it The eval lifecycle (analyze → measure → improve) Error analysis — the hardest step: take traces and find failure modes. No perfect definition of “mistake” (you can’t define slop, but you know it when you see it) Measure — how prevalent is each failure mode? Pareto applies: ~80% of issues come from ~20% of failure modes — prioritize those Improve — fix the product: prompt instructions, model switch, fine-tuning. Iterate forever AI is weak at the front (taste-specific error analysis) and strong at the back (measurement, prompt optimization, hill-climbing). ...

Hamel Husain, a decade-plus AI consultant himself, argues the hottest job in the economy is usually a bad hire — with two exceptions. ~3 minutes on his own channel. Why hiring an AI consultant usually fails The problems you had before — talent, organizational, skill issues — are still there when the consultant leaves AI’s whole point is to remove the middleman: you express what you want in natural language and inject your domain knowledge directly into the tool If they just implement and walk away, you can’t maintain what they built — worst case you keep paying them, i.e. prompting a consultant instead of prompting an AI His track record: in over a decade, third-party AI implementations he’s seen don’t work When a consultant is worth it Education: the consultant up-levels your entire team and makes themselves obsolete — you keep the skills after they’re gone Skin in the game: they don’t get paid unless they affect your bottom line (increase revenue or decrease costs), demonstrated at a quality bar — aligned incentives The alternative Learn AI yourself — it’s the most accessible technology ever created; you mostly need to learn how to talk to it Coding skills help but aren’t required for lots of things; a growth mindset makes effective use tractable with practice “Instead of prompting an AI, you don’t want to be prompting a consultant.” ...

Hamel Husain answers a client question — how to eval an AI PE curriculum planner for K-12 — with a one-minute twist: evals and product trust are the same problem. ~1 minute on his own channel. The eval problem A client building an AI curriculum planner for K-12 physical education couldn’t figure out how to eval it The answer: think creatively — put yourself in the educator’s shoes Questions an educator actually asks Was this plan created by an expert? What other teachers are using this plan? Is this plan based on best practices and expert knowledge? The takeaway Infuse those signals into the product — they give the end user trust in what they’re seeing That same design doubles as your eval: provide users a way to assess what they see, and you can assess it too If you can’t eval your product, your product probably sucks “If you can’t eval your product, your product probably sucks.” ...

Hamel Husain on his own channel — a 1-minute clip on the “evals take too long” objection and why your users eval your product whether you do or not. The “evals take too long” objection One of the main reasons people skip evals: reading a trace takes forever — a lot of human hours you “don’t have time to do” His example: an AI data agent that answers business questions (supply, revenue) — “the most popular internal tool I’ve seen so far” The question: how do you eval that? If you can’t eval it, it probably sucks One principle to keep in mind: if you can’t eval your product, your product probably sucks — because your users need to eval the product When you present an AI answer to a user — say, an answer to a business question — you need to figure out a way to convey trust “If you can’t eval your product, your product probably sucks. Why is that? It’s because your users need to eval the product.” — Hamel Husain ...

Hamel Husain on his own channel — a 1.5-minute clip on LLM-as-a-judge evals: how to measure the judge’s noise and why a judge you can trust 100% of the time doesn’t exist. Measure the judge against ground truth An LLM judge is a “soft check” — you can scientifically measure how good it is by comparing it to labels Assemble a dataset of human labels as ground truth (is this thing good or bad?) and measure how much noise the judge has Even ~100 labels is enough to measure the noise There are different kinds of noise — you can tune the judge to reduce it, but there will always be some A judge is a black-box classifier There’s no such thing as a judge you can blindly trust with no noise at all It’s a classifier, like a stop-sign detector: get it very reliable and there’s still an edge case somewhere producing false positives and false negatives You have to be okay with that Tolerable noise is a business decision Tune the judge to reduce noise, then make the call: is this level of noise tolerable for your use case? Bake the eval’s known characteristics into your decision making — that’s the best you can do with an LLM judge “Even if you can get that to be very reliable, there’ll always be an edge case somewhere where it’s going to give you a false positive, false negative.” — Hamel Husain ...

Hamel Husain on his own channel — a 2-minute clip on why traditional scoring metrics (ROUGE/BLEU) and rating scales make evals harder to act on, and why he defaults to binary pass/fail. ROUGE/BLEU: string similarity from a different era These scores come from traditional ML/NLP — at their core they measure string similarity A coarse metric that made sense when LLMs could barely produce coherent language; “we’re way past the point of producing coherent English” now Part of the “eval industrial complex”: we have metrics, trust us because we’re experts — it feels fancy but isn’t what you need Rating scales hide uncertainty Unless you have a very sophisticated setup and put real resources into aligning scores with humans — which ~99% of teams don’t — a 1-to-10 or 1-to-5 scale is a bad idea Nobody knows what 4.2 versus 3.7 means Humans just hide their uncertainty in the middle values, so you don’t get good decision making out of it Binary pass/fail is what you can ship on At the end of the day you have to ship your product: is this good or not? Binary evals are easier to align with humans — which you always have to do with an eval — and easier to action A score of 3.2 → “I don’t know what’s wrong.” A fail → “okay, it failed, now you can action on that” “You see a score of 3.2, you’re like, I don’t know what’s wrong… but fail, it’s like, okay, it failed, now you can action on that.” — Hamel Husain ...

Hamel Husain on his own channel — a 40-second clip on why rating-scale evals (1–5) are the wrong default: he pushes for binary pass/fail evals with failures scoped into specific, actionable criteria. Binary beats a 1–5 scale He tries “really hard” to make every eval binary — pass or fail Scope the failure: “agent took too long → failed because it took too long”, “too many steps → failed because it took too many steps” He has never gotten stuck converting a 1–5 scale into binary, across all the companies he’s worked with The exception: a well-tested rubric everyone is confident in — then a 1–5 scale might genuinely work In most cases the scale just “kicks the can down the road” — the noise doesn’t disappear, it gets hidden “In a lot of cases, it just kind of kicks the can down the road, and you’re just hiding a lot of noise in there.” — Hamel Husain ...