Llms

Joe Barrow (ex-Amazon/Adobe, ML lead at Pattern Data, now Adobe Research’s Document Intelligence Lab) on choosing an OCR model for AI document processing. 24 minutes on Hamel Husain’s channel. Why OCR matters Your app sits downstream of OCR quality — garbage in, garbage out, no matter what the LLM does after It’s not solved — Anthropic shipped bad PDF handling for a year because it was pulling text, not doing OCR; users noticed OCR is sticky — once you build on a vendor, swapping models is painful (Pattern learned this the hard way) Documents are evil — multi-column layouts, rotated scans, no reading order; TeX-compiled PDFs have no spaces (glyph glue), so naive text extraction gives you one run of characters The decision grid: two axes Text blocks vs. document structure Text blocks: word/line bounding boxes → grounding, evidence highlighting, cheapest Structure: headings, reading order, grouped paragraphs, tables, figure alt text, chart de-rendering → much better LLM input (LLMs are trained on markdown-like structure; raw line runs look like garbage to them) API vs. self-host API: ease of use, vendor support (startups retrain on your bad docs), minimal time — right for ~95% of teams Self-host: control throughput/concurrency (APIs cap concurrent docs — a real bottleneck), stable weights, domain fine-tuning, no lock-in, cheaper at bulk — but only if your time ≈ $0 or you run huge batches The four quadrants Big cloud APIs (AWS Textract, Google Cloud Vision, Azure) — $0.60–1.50 / 1k pages; word+line boxes only; tables/forms a la carte at $10–15 / 1k Document startups (Reducto, Data Lab, Extend, LlamaIndex) — $5–20 / 1k pages, “fast” vs “accurate” tiers; structure included (markdown/HTML, tables, figure boxes) Open pipelines (PaddleOCR, Nemo Tron, Tesseract) — 10–100M params, nearly free, edge-deployable (PaddleOCR runs on phones/e-ink); text lines only, post-process with layout models Open VLMs (LightOn OCR 2, GLM OCR, GOT-OCR, Chandra/Surya) — 600M–8B params, native document structure, ~$0.20–0.30 / 1k pages on a saturated H100; hallucination risk exists but clouds hallucinate on crusty scans too How to actually choose Ignore benchmarks (OmniDocBench, CR Bench) — they’re not run on your data Build a 50–100 page sample of your own representative PDFs Run a few candidates, diff the returned text (catches junk-on-handwriting fast), visualize the boxes ~a day of effort total — then pick Watch the license Chandra/Surya (Data Lab): free only if org < $2M revenue AND not competing with Data Lab LightOn OCR: Apache. GLM OCR: MIT (but relies on PaddlePaddle’s Doc Layout model — Apache — both apply) Self-hosting, for the ~5% who should Inference engines: VL (default, OpenAI-style client) or SGLang; infra: Modal (request-queue scaling beats SageMaker), BaseTen, or big cloud for one-off batches 1B-param models (LightOn, GLM) on H100 → ~10k pages/hr, 20–30¢ / 1k pages; 4×3090 ≈ one H100 → 3–4 pages/sec His 7M-page local-laws dataset: ran over a weekend at ~30¢ / 1k all-in “You can process 1,000 pages per second, but it doesn’t matter if they’re all wrong — then your entire app’s output is going to be garbage.” ...

Jeff Su’s counter to the default “jump in, pick a template, start prompting” tutorials — that path gives you generic output and burns tokens fixing unusable slides. His fix: three files prepared ahead of time, demonstrated with the actual deck he used for a paid workshop. ...

Futurepedia’s full-platform guide to Claude Design after its big upgrade — the host skipped covering it at launch because usage limits made it barely usable; that’s fixed (usage now bundles into your existing Claude credits). The overview: 15+ template types (mobile apps, slides, documents, wireframes, animations, UI mockups, resumes, 3D objects, HTML email, flyers), and the design-system workflow that stops output from looking like generic AI slop. ...

Lucas Rocha, an engineer at Brazilian edtech Nova Escola and alum of Hamel Husain’s AI Evals course, tells his evals rollout story backwards on Hamel’s channel — 34 minutes on going from a messy spreadsheet to four calibrated judges running in CI and on daily production samples. ...

Abi Aryan (“Inference Infra Queen”, Maven inference-engineering instructor, NOVA IMS) teaches LLM latency from first principles on Hamel Husain’s channel — a 41-minute Colab demo that separates prefill from decode and shows why decode owns your latency and cost. Where the time actually goes Opening puzzle: the same TinyLlama model, same T4 GPU, same inference code produced one request at 3.3s and another at 8.8s — no tricks, no patching Most people’s mental model (“more tokens = more compute”) is wrong; the request shape decides where time goes The inference shape = 4 variables: input tokens, output tokens, batch size (concurrent requests), context length (what attention can see) Real workloads cluster into 4 shapes: chat (~1:1), RAG (~20:1 input:output), creative writing (tiny prompt, long output), agentic (no fixed ratio — the hardest to predict) Agentic shapes are global, not static: the request at step 6 contains every token from steps 1–5 Prefill vs decode: two different bottlenecks Prefill: whole prompt in one parallel forward pass, produces the KV cache; compute-bound and GPU-saturating; scales with input tokens; experienced as time-to-first-token Decode: autoregressive, one token at a time; memory-bandwidth-bound — every token streams the full weights + growing KV cache (2.2GB of HBM traffic for one TinyLlama token); scales with output tokens; experienced as streaming speed “A GPU is just not another faster CPU” — the two phases have different bottlenecks, so different optimization strategies TTFT and streaming speed are reported as separate numbers because they come from different phases The Colab demo: plain Transformers, no engine Deliberately runs Hugging Face Transformers in eager mode — no vLLM/SGLang — because vLLM hides the prefill/decode split (and doesn’t install cleanly on Colab); seeing the raw loop first is what makes the engines legible TinyLlama 1.1B in FP16 = 2.2GB VRAM on a T4; the inference function syncs to the GPU clock (torch.cuda.synchronize) and times prefill vs decode per request Smoke test first (22 in / 8 out tokens): prefill 152ms ≈ 144 tok/s, decode 7 tok/s — same model, same GPU, different phase, different bottleneck Always warm up the GPU before experiments — skipping warmup in production is a classic footgun The three shapes, measured Short-in/short-out (chat): prefill ~40ms, decode dominates — prefill ≈ 3% of latency Long-in/short-out (RAG): prefill share grows to ~16% — the only shape where prefill visibly matters Short-in/long-out (creative writing): prefill ≈ 0.5% — and this shape has the longest total latency, because decode is sequential Read vs write asymmetry: ~1,205 input tokens prefill at ~0.44ms/token — writing a token costs roughly 300x more GPU-clock time than reading one That’s why pricing pages price input and output tokens separately Small requests waste the GPU Prefill throughput: 143 tok/s (25-token prompt) vs 2,271 tok/s (long prompt) — a 16x difference in the same phase A tiny request leaves the GPU nearly idle — the observation that birthed continuous batching (vLLM/SGLang’s scheduler.py batches as many prefill tokens as possible into one request) KV cache: growth depends on architecture KV cache = memory growing with output tokens; ~0.17MB per token for TinyLlama, measured under theory Growth is architecture-dependent: TinyLlama uses GQA (KV pairs shared across a group of heads); MQA/MLA compress differently; MoE models (e.g. DeepSeek) generate less KV cache When picking a model, prefer architectures that group the KV cache — “think about this as the context filling” Agentic loops: where cost really lands Simulated 5-step agent loop (build prompt → inference → append to history), no framework — frameworks add latency, hide token consumption, and fail fast in production Every step re-sends the full history from scratch (no context graph, no memory card) — runtime grows ~O(n²) in input tokens Result: prefill ≈ 3% of agentic latency; decode accumulates every step; cumulative latency hugs the decode line Agent shapes add three more variables: batch size, context window, number of tool calls (more tool calls → more latency) Costed on OpenAI pricing: a 5-step loop ran up 11.9x the cost of one chat request — almost all from re-sent input tokens feeding decode The agent-optimization target: the smallest amount of KV cache you can share back to do the decode The dense takeaways Latency lives in decode; output tokens are ~2 orders of magnitude more expensive than input tokens TTFT is a prefill story; total latency is an output-length story — cap max tokens, don’t just tune the retrieval pipeline Small requests waste the GPU — batch (143 → 2,271 tok/s) or let the engine’s scheduler do it Agents rebuy their entire history every step — context management is survival, not optimization Homework Try batch sizes 4 and 8 vs 1 and watch prefill/decode latency change Swap engines: llama.cpp on Mac (auto-batching/scheduling), vLLM/SGLang on Linux — how does each affect the phases? Try INT8/INT4 quantized versions of the same FP16 model — memory, latency, cost per phase Explore prefix caching: smallest KV cache you can hand back; read vLLM’s batch.py and the paged/flash attention kernels Q&A: production mistakes and scaling Mistake #1: shipping models with no smoke test/warmup before pods report ready Mistake #2: serving the same model on different ports without KV-cache sharing across them Open problem: how long to hold KV cache before eviction — dead weight, the same question database companies faced Self-hosting economics: hybrid approach — small one-off requests on API GPUs, long-running multi-step agentic systems on your own GPUs (EU regulation is the deciding factor in Europe) Speculative decoding & chunked prefill: built to control KV-cache growth on very large models (~45B+); not a major concern for small ones Resources: her Maven courses (forward-deployed; inference engineering & systems design), gpengineering.com with her PMPP notes, plus intro and kernel-engineering books “Context management… that’s what will make the difference between whether you’re building a system that would survive or whether you’re building a system that will eventually get killed.” — Abi Aryan, on agentic systems ...

Hamel Husain makes the case that “are you building the right thing?” matters more than evals: if users can’t verify an AI answer, your product creates work instead of saving it. Three worked examples and four design principles. ~17 minutes on his own channel. The core problem: verification is the bottleneck Classic AI product: user asks “what was revenue for product A last quarter?”, AI queries a database and returns “$4.21M” — but the user can’t validate the number without redoing all the work Lenny’s tweet: data science teams now spend their time reviewing half-assed AI analysis from PMs and data engineers — 50% of the time it’s wrong Creating output is now easy; verification is the bottleneck — design products with that in mind If a product seems impossible to eval, that’s a product smell: your users have to check it too Example 1: business Q&A → evidence-backed analysis Instead of a bare answer, show supporting information: base the analysis on already-vetted analysis (notebooks others signed off on) State the assumptions — confirm the metric definition (from a semantic layer, linkable to the governed definition) Show intermediate calculations (returns, customer counts) and raise issues the user might care about “Open as notebook” UI idea: an AI-generated notebook (Jupyter/Marimo style) with narrative + queries — step through it, edit it, ask AI questions about it, and have the AI state what it couldn’t verify Put yourself in the expert’s shoes: a data scientist checks prior analyses, definitions, and intermediate calcs — give users those same affordances Real product doing this today: Hex — a chat interface that flows into a notebook; progressive disclosure is an established pattern we forgot in AI apps Example 2: PE lesson plan assistant (K-12) Vanilla version: input grade, class length, location, equipment → AI spits out a lesson plan — no way to judge quality Better: anchor the output in existing curriculum — what are trusted colleagues doing? “Here’s a lesson plan exactly like yours, used at 14 schools, run 30 times, by teachers you trust” Show the edits made for your inputs (“shortened to a 45-minute class, matched to your equipment”) — accept or reject each Cold-start problem: seed with expert-vetted plans up front; it makes the app easier to eval and users more confident Example 3: medical claims report generator The task: read a patient’s chart + claim (thousands of documents) and produce a 52-page report supporting or denying a claim — a doctor can’t eyeball 52 pages Redesign as a research assistant: surface atomic elements first — contradictions (open both pages, resolve), key facts (validate, include, dismiss), open questions (add notes) Do the work the user would normally do and guide them through it; understanding built along the way is arguably more valuable than the final report Four design principles Provenance — show where information comes from (prior analyses, source documents, other lesson plans); the more curated/trustworthy, the better Progressive disclosure — show details at the right time; adapt as your product and users mature Signals and heuristics — mirror how experts sanity-check: other reports, smell tests, social proof, supporting evidence Modularity / small steps — break work into small verifiable pieces, show your work along the way, and sometimes force the human in the loop (e.g. doctors step through findings before the final report) “The goal is to make the human understand, not necessarily to generate a report.” ...

Shreya Shankar (Stanford CS professor, co-creator of the AI evals course with Hamel) kicks off the 12-part AI product engineering series. 27 minutes on Hamel Husain’s channel. Why this matters Output quality is the biggest barrier to productionizing agents (LangSmith annual report) — and figuring out how to evaluate models is genuinely hard Vendors (LangChain, Braintrust, Arize) are selling end-to-end automated eval tools: point an LLM at your traces, it finds and fixes your bugs The catch is epistemic: what “good” means lives in your head, not in the traces — if a tool could fully fix your product, it could fix everyone’s, and there’d be nothing left to differentiate yours AI’s real job: help you express and apply your judgment faster, not replace it The eval lifecycle (analyze → measure → improve) Error analysis — the hardest step: take traces and find failure modes. No perfect definition of “mistake” (you can’t define slop, but you know it when you see it) Measure — how prevalent is each failure mode? Pareto applies: ~80% of issues come from ~20% of failure modes — prioritize those Improve — fix the product: prompt instructions, model switch, fine-tuning. Iterate forever AI is weak at the front (taste-specific error analysis) and strong at the back (measurement, prompt optimization, hill-climbing). ...

Hamel Husain on his own channel — a 1.5-minute clip on LLM-as-a-judge evals: how to measure the judge’s noise and why a judge you can trust 100% of the time doesn’t exist. Measure the judge against ground truth An LLM judge is a “soft check” — you can scientifically measure how good it is by comparing it to labels Assemble a dataset of human labels as ground truth (is this thing good or bad?) and measure how much noise the judge has Even ~100 labels is enough to measure the noise There are different kinds of noise — you can tune the judge to reduce it, but there will always be some A judge is a black-box classifier There’s no such thing as a judge you can blindly trust with no noise at all It’s a classifier, like a stop-sign detector: get it very reliable and there’s still an edge case somewhere producing false positives and false negatives You have to be okay with that Tolerable noise is a business decision Tune the judge to reduce noise, then make the call: is this level of noise tolerable for your use case? Bake the eval’s known characteristics into your decision making — that’s the best you can do with an LLM judge “Even if you can get that to be very reliable, there’ll always be an edge case somewhere where it’s going to give you a false positive, false negative.” — Hamel Husain ...

An 85-minute conversation between Dan Shipper (Every) and Paul Ford — co-founder of Abort/Aboard, former Harper’s editor, author of the Bloomberg classic “What Is Code?” — recorded days after Opus 4.5 landed inside Claude Code, at a moment both describe as “the world changed last week.” Two deeply reflective practitioners trying to metabolize a genuine step change in real time. The step change Opus 4.5 is the first time vibe coding “just keeps going without tripping over itself” — it builds, and fixes its own errors Paul built a fully-featured iPhone reading app (photo → analysis → research agent → custom reading profile) with no idea how it works Paul’s framing: NOT a 9,000x model jump — a product step change. Claude Code added agent-style self-evaluation. It’s “the first true product built on top of an LLM” Claude Code’s design principle Anything you can do on your computer, Claude Code can do — low-level tools (files, grep, bash) below the level of features Features are just prompts: slash commands and subagents, writable in English General product principle: move what used to be code functionality into prompts the agent executes with low-level tools The emerging skill: abstraction-level thinking Don’t hand the agent the problem — hand it the way to get information about the problem, then constraints Paul’s synth pipeline: spider DSP textbooks into a SQLite reference → constrain to good open-source libraries → implement. Five or six levels up, and “make me a synth like this” works “That’s the skill that’s going to be emerging” The hard truths “I no longer feel I can in good faith say human skills are going to be relevant” — 600K jobs at Accenture alone, 50M devs worldwide “Everyone gets the same Pokemon shoved into the mailbox” — the power is universal, instantly The GLP-1 analogy: rules of a lifetime can change overnight, and a year or two is nowhere near enough to metabolize it “Software was eating the world. Now it’s eating itself.” His concept: latent software — the PDFs and spreadsheets that describe software that doesn’t exist yet The discourse taxonomy AGI-is-coming group: gone quiet because there’s money to be made. Sam Altman “wants to be Steve Jobs but he’s Steve Ballmer.” OpenAI is Microsoft; Anthropic is Google. Nobody is Apple — “you can’t put a civilian in front of that interface” Left-adjacent literary types (his Harper’s world): want their prose untouched Rejecters vs. do-gooders: charities and climate scientists can’t wait to use it to accelerate missions that are “unalloyed good” Professors who keep it away from students: he respects that line completely The real harms Provenance: “I want nutritional guidelines for what’s in my Anthropic LLM” — Google honors robots.txt; LLMs don’t tell you what’s in them The devaluation of the 50M-person underpinning of the global economy — “who gets to talk about that?” The failure to plan: “people see it coming but don’t really plan for it” The Sankey chart Paul had Claude build a mild-bearish model of consulting’s future: McKinsey $16B → $4B by 2035; Alexander makes partner in 2029 “just as the firm started its long contraction. She was one of the last… the smartest thing in every room now was the computer.” Shared with a consultant: “they got quiet for a minute. And they went, ‘interesting.’” ...

Horace He (Thinking Machines Lab) on one of the most annoying facts about LLMs: ask the same model the same question twice and you can get different answers — even at temperature 0, where the math says the model should always pick the same token. The common explanation is wrong The usual suspect: “concurrency + floating point” — GPU threads racing to accumulate results in different orders, so the same kernel gives different answers run to run But the same matrix multiplication on the same data is bitwise identical 1000 times in a row. GPUs are definitely concurrent and floating point is definitely involved — so that’s not the whole story The real foundation is floating-point non-associativity: (a+b)+c ≠ a+(b+c). Adding numbers in a different order genuinely changes the result. The question is what changes the order The actual culprit: your batch size Four things are simultaneously true: some GPU kernels are nondeterministic; every kernel in an LLM’s forward pass is deterministic; the inference server is deterministic; and yet users see nondeterminism. ...