Posts

Hamel Husain demos a Codex capability most people don’t know about: Codex can control itself — spawning parallel threads, letting them coordinate, and driving computer use. ~5 minutes on his own channel. The demo setup Working project: AEO — optimizing one of his websites for AI discovery (making it findable by AI search engines) Codex listed 16 high-value tasks from the project file (aeo-to-do.md) — the fuel for the orchestration demo Spawning threads Prompt: “Open a new thread for each task and explain how you’d tackle it, along with prerequisite steps” — Codex spawns 16 parallel threads in the sidebar Codex can also rename and delete threads itself Value: manage separate tasks completely independently, no window-jumping Threads talking to threads Inside any thread you can query another: “What is AEO 1 doing? Does it need any help?” Great for orchestrating when things get stuck, or starting a supervisor thread that manages others and unblocks them Steering and queues Ask for a status table when threads finish: which can run in parallel, which need human intervention or input Broadcast guidance to all threads: “Direct threads that can work independently with computer use to start — don’t start work if you need other threads to finish first” One thread inventories the active threads and coordinates the rest — “this starts to become super powerful” Computer use A thread opens the browser itself: checks Bing Webmaster Tools, Google Search Console, etc. It tells Hamel what it needs (accepting a verification), keeps going, and reports when it gets stuck Mobile Same thread list appears on your phone — manage all parallel threads remotely, even away from the computer “You can have Codex control Codex and become a power user to do a lot of things faster and parallelize your work.” ...

Joe Barrow (ex-Amazon/Adobe, ML lead at Pattern Data, now Adobe Research’s Document Intelligence Lab) on choosing an OCR model for AI document processing. 24 minutes on Hamel Husain’s channel. Why OCR matters Your app sits downstream of OCR quality — garbage in, garbage out, no matter what the LLM does after It’s not solved — Anthropic shipped bad PDF handling for a year because it was pulling text, not doing OCR; users noticed OCR is sticky — once you build on a vendor, swapping models is painful (Pattern learned this the hard way) Documents are evil — multi-column layouts, rotated scans, no reading order; TeX-compiled PDFs have no spaces (glyph glue), so naive text extraction gives you one run of characters The decision grid: two axes Text blocks vs. document structure Text blocks: word/line bounding boxes → grounding, evidence highlighting, cheapest Structure: headings, reading order, grouped paragraphs, tables, figure alt text, chart de-rendering → much better LLM input (LLMs are trained on markdown-like structure; raw line runs look like garbage to them) API vs. self-host API: ease of use, vendor support (startups retrain on your bad docs), minimal time — right for ~95% of teams Self-host: control throughput/concurrency (APIs cap concurrent docs — a real bottleneck), stable weights, domain fine-tuning, no lock-in, cheaper at bulk — but only if your time ≈ $0 or you run huge batches The four quadrants Big cloud APIs (AWS Textract, Google Cloud Vision, Azure) — $0.60–1.50 / 1k pages; word+line boxes only; tables/forms a la carte at $10–15 / 1k Document startups (Reducto, Data Lab, Extend, LlamaIndex) — $5–20 / 1k pages, “fast” vs “accurate” tiers; structure included (markdown/HTML, tables, figure boxes) Open pipelines (PaddleOCR, Nemo Tron, Tesseract) — 10–100M params, nearly free, edge-deployable (PaddleOCR runs on phones/e-ink); text lines only, post-process with layout models Open VLMs (LightOn OCR 2, GLM OCR, GOT-OCR, Chandra/Surya) — 600M–8B params, native document structure, ~$0.20–0.30 / 1k pages on a saturated H100; hallucination risk exists but clouds hallucinate on crusty scans too How to actually choose Ignore benchmarks (OmniDocBench, CR Bench) — they’re not run on your data Build a 50–100 page sample of your own representative PDFs Run a few candidates, diff the returned text (catches junk-on-handwriting fast), visualize the boxes ~a day of effort total — then pick Watch the license Chandra/Surya (Data Lab): free only if org < $2M revenue AND not competing with Data Lab LightOn OCR: Apache. GLM OCR: MIT (but relies on PaddlePaddle’s Doc Layout model — Apache — both apply) Self-hosting, for the ~5% who should Inference engines: VL (default, OpenAI-style client) or SGLang; infra: Modal (request-queue scaling beats SageMaker), BaseTen, or big cloud for one-off batches 1B-param models (LightOn, GLM) on H100 → ~10k pages/hr, 20–30¢ / 1k pages; 4×3090 ≈ one H100 → 3–4 pages/sec His 7M-page local-laws dataset: ran over a weekend at ~30¢ / 1k all-in “You can process 1,000 pages per second, but it doesn’t matter if they’re all wrong — then your entire app’s output is going to be garbage.” ...

Jeff Su’s counter to the default “jump in, pick a template, start prompting” tutorials — that path gives you generic output and burns tokens fixing unusable slides. His fix: three files prepared ahead of time, demonstrated with the actual deck he used for a paid workshop. ...

Futurepedia’s full-platform guide to Claude Design after its big upgrade — the host skipped covering it at launch because usage limits made it barely usable; that’s fixed (usage now bundles into your existing Claude credits). The overview: 15+ template types (mobile apps, slides, documents, wireframes, animations, UI mockups, resumes, 3D objects, HTML email, flyers), and the design-system workflow that stops output from looking like generic AI slop. ...

Lucas Rocha, an engineer at Brazilian edtech Nova Escola and alum of Hamel Husain’s AI Evals course, tells his evals rollout story backwards on Hamel’s channel — 34 minutes on going from a messy spreadsheet to four calibrated judges running in CI and on daily production samples. ...

Matt Pocock live-demos Wayfinder — the evolution of his Grill Me skill, for the pre-spec stage of software work. Wayfinder charts a “map” from the current codebase to a destination (a locked set of decisions / a spec), breaks a foggy requirement into parallel tickets, and spawns agents to work them. The demo builds a TikTok creator feature for his Course Video Manager app — 8 of 9 map tasks in ~75 minutes, without writing implementation code. ...

Abi Aryan (“Inference Infra Queen”, Maven inference-engineering instructor, NOVA IMS) teaches LLM latency from first principles on Hamel Husain’s channel — a 41-minute Colab demo that separates prefill from decode and shows why decode owns your latency and cost. Where the time actually goes Opening puzzle: the same TinyLlama model, same T4 GPU, same inference code produced one request at 3.3s and another at 8.8s — no tricks, no patching Most people’s mental model (“more tokens = more compute”) is wrong; the request shape decides where time goes The inference shape = 4 variables: input tokens, output tokens, batch size (concurrent requests), context length (what attention can see) Real workloads cluster into 4 shapes: chat (~1:1), RAG (~20:1 input:output), creative writing (tiny prompt, long output), agentic (no fixed ratio — the hardest to predict) Agentic shapes are global, not static: the request at step 6 contains every token from steps 1–5 Prefill vs decode: two different bottlenecks Prefill: whole prompt in one parallel forward pass, produces the KV cache; compute-bound and GPU-saturating; scales with input tokens; experienced as time-to-first-token Decode: autoregressive, one token at a time; memory-bandwidth-bound — every token streams the full weights + growing KV cache (2.2GB of HBM traffic for one TinyLlama token); scales with output tokens; experienced as streaming speed “A GPU is just not another faster CPU” — the two phases have different bottlenecks, so different optimization strategies TTFT and streaming speed are reported as separate numbers because they come from different phases The Colab demo: plain Transformers, no engine Deliberately runs Hugging Face Transformers in eager mode — no vLLM/SGLang — because vLLM hides the prefill/decode split (and doesn’t install cleanly on Colab); seeing the raw loop first is what makes the engines legible TinyLlama 1.1B in FP16 = 2.2GB VRAM on a T4; the inference function syncs to the GPU clock (torch.cuda.synchronize) and times prefill vs decode per request Smoke test first (22 in / 8 out tokens): prefill 152ms ≈ 144 tok/s, decode 7 tok/s — same model, same GPU, different phase, different bottleneck Always warm up the GPU before experiments — skipping warmup in production is a classic footgun The three shapes, measured Short-in/short-out (chat): prefill ~40ms, decode dominates — prefill ≈ 3% of latency Long-in/short-out (RAG): prefill share grows to ~16% — the only shape where prefill visibly matters Short-in/long-out (creative writing): prefill ≈ 0.5% — and this shape has the longest total latency, because decode is sequential Read vs write asymmetry: ~1,205 input tokens prefill at ~0.44ms/token — writing a token costs roughly 300x more GPU-clock time than reading one That’s why pricing pages price input and output tokens separately Small requests waste the GPU Prefill throughput: 143 tok/s (25-token prompt) vs 2,271 tok/s (long prompt) — a 16x difference in the same phase A tiny request leaves the GPU nearly idle — the observation that birthed continuous batching (vLLM/SGLang’s scheduler.py batches as many prefill tokens as possible into one request) KV cache: growth depends on architecture KV cache = memory growing with output tokens; ~0.17MB per token for TinyLlama, measured under theory Growth is architecture-dependent: TinyLlama uses GQA (KV pairs shared across a group of heads); MQA/MLA compress differently; MoE models (e.g. DeepSeek) generate less KV cache When picking a model, prefer architectures that group the KV cache — “think about this as the context filling” Agentic loops: where cost really lands Simulated 5-step agent loop (build prompt → inference → append to history), no framework — frameworks add latency, hide token consumption, and fail fast in production Every step re-sends the full history from scratch (no context graph, no memory card) — runtime grows ~O(n²) in input tokens Result: prefill ≈ 3% of agentic latency; decode accumulates every step; cumulative latency hugs the decode line Agent shapes add three more variables: batch size, context window, number of tool calls (more tool calls → more latency) Costed on OpenAI pricing: a 5-step loop ran up 11.9x the cost of one chat request — almost all from re-sent input tokens feeding decode The agent-optimization target: the smallest amount of KV cache you can share back to do the decode The dense takeaways Latency lives in decode; output tokens are ~2 orders of magnitude more expensive than input tokens TTFT is a prefill story; total latency is an output-length story — cap max tokens, don’t just tune the retrieval pipeline Small requests waste the GPU — batch (143 → 2,271 tok/s) or let the engine’s scheduler do it Agents rebuy their entire history every step — context management is survival, not optimization Homework Try batch sizes 4 and 8 vs 1 and watch prefill/decode latency change Swap engines: llama.cpp on Mac (auto-batching/scheduling), vLLM/SGLang on Linux — how does each affect the phases? Try INT8/INT4 quantized versions of the same FP16 model — memory, latency, cost per phase Explore prefix caching: smallest KV cache you can hand back; read vLLM’s batch.py and the paged/flash attention kernels Q&A: production mistakes and scaling Mistake #1: shipping models with no smoke test/warmup before pods report ready Mistake #2: serving the same model on different ports without KV-cache sharing across them Open problem: how long to hold KV cache before eviction — dead weight, the same question database companies faced Self-hosting economics: hybrid approach — small one-off requests on API GPUs, long-running multi-step agentic systems on your own GPUs (EU regulation is the deciding factor in Europe) Speculative decoding & chunked prefill: built to control KV-cache growth on very large models (~45B+); not a major concern for small ones Resources: her Maven courses (forward-deployed; inference engineering & systems design), gpengineering.com with her PMPP notes, plus intro and kernel-engineering books “Context management… that’s what will make the difference between whether you’re building a system that would survive or whether you’re building a system that will eventually get killed.” — Abi Aryan, on agentic systems ...

Hamel Husain makes the case that “are you building the right thing?” matters more than evals: if users can’t verify an AI answer, your product creates work instead of saving it. Three worked examples and four design principles. ~17 minutes on his own channel. The core problem: verification is the bottleneck Classic AI product: user asks “what was revenue for product A last quarter?”, AI queries a database and returns “$4.21M” — but the user can’t validate the number without redoing all the work Lenny’s tweet: data science teams now spend their time reviewing half-assed AI analysis from PMs and data engineers — 50% of the time it’s wrong Creating output is now easy; verification is the bottleneck — design products with that in mind If a product seems impossible to eval, that’s a product smell: your users have to check it too Example 1: business Q&A → evidence-backed analysis Instead of a bare answer, show supporting information: base the analysis on already-vetted analysis (notebooks others signed off on) State the assumptions — confirm the metric definition (from a semantic layer, linkable to the governed definition) Show intermediate calculations (returns, customer counts) and raise issues the user might care about “Open as notebook” UI idea: an AI-generated notebook (Jupyter/Marimo style) with narrative + queries — step through it, edit it, ask AI questions about it, and have the AI state what it couldn’t verify Put yourself in the expert’s shoes: a data scientist checks prior analyses, definitions, and intermediate calcs — give users those same affordances Real product doing this today: Hex — a chat interface that flows into a notebook; progressive disclosure is an established pattern we forgot in AI apps Example 2: PE lesson plan assistant (K-12) Vanilla version: input grade, class length, location, equipment → AI spits out a lesson plan — no way to judge quality Better: anchor the output in existing curriculum — what are trusted colleagues doing? “Here’s a lesson plan exactly like yours, used at 14 schools, run 30 times, by teachers you trust” Show the edits made for your inputs (“shortened to a 45-minute class, matched to your equipment”) — accept or reject each Cold-start problem: seed with expert-vetted plans up front; it makes the app easier to eval and users more confident Example 3: medical claims report generator The task: read a patient’s chart + claim (thousands of documents) and produce a 52-page report supporting or denying a claim — a doctor can’t eyeball 52 pages Redesign as a research assistant: surface atomic elements first — contradictions (open both pages, resolve), key facts (validate, include, dismiss), open questions (add notes) Do the work the user would normally do and guide them through it; understanding built along the way is arguably more valuable than the final report Four design principles Provenance — show where information comes from (prior analyses, source documents, other lesson plans); the more curated/trustworthy, the better Progressive disclosure — show details at the right time; adapt as your product and users mature Signals and heuristics — mirror how experts sanity-check: other reports, smell tests, social proof, supporting evidence Modularity / small steps — break work into small verifiable pieces, show your work along the way, and sometimes force the human in the loop (e.g. doctors step through findings before the final report) “The goal is to make the human understand, not necessarily to generate a report.” ...

Hamel Husain demos the Codex desktop app’s remote-control mode — the best setup he’s found for untethering AI coding from your computer and running sessions from your phone. ~4 minutes on his own channel. Why remote control When coding with AI, don’t be tethered to your laptop — walk around, close the screen, check things on your phone The Codex desktop app is the best client he’s found for remote AI coding Setup: connect your computers Settings → Connections — he’s connected to a Mac mini and a MacBook Pro (both servers for agentic coding) To add: go to the server, click the link to get a code, then Add it in the app Keep the connection dots green — refresh occasionally; rarely you may need to delete and re-add a connection Keep the Codex app updated (Codex menu → check for updates) Starting a remote session New chat (Cmd+N) or Add new project → Remote, then pick the connected computer and give the folder path You see all sessions running remotely — which machine, which folder, live green dot Live demo Prompt: “Look at my Gmail and find the last 10 emails received from this email address, which are newsletter posts. Put it in the table” — uses the Gmail skill/connector and computer use on the MacBook Pro From the phone (ChatGPT app → Codex tab): see every connected computer and live session; send follow-ups like “Please critique my writing in all 10 emails with subagents. Use the writing skills in this repo to do so” Desktop shows the run: 5.5 High extra fast model, summary view listing the 4 spawned subagents processing email batches; phone shows 2 agents running / 2 finished You can even attach photos or screenshots from your phone “It’s the best remote control I’ve ever used. I haven’t found any limitations.” ...

Shreya Shankar (Stanford CS professor, co-creator of the AI evals course with Hamel) kicks off the 12-part AI product engineering series. 27 minutes on Hamel Husain’s channel. Why this matters Output quality is the biggest barrier to productionizing agents (LangSmith annual report) — and figuring out how to evaluate models is genuinely hard Vendors (LangChain, Braintrust, Arize) are selling end-to-end automated eval tools: point an LLM at your traces, it finds and fixes your bugs The catch is epistemic: what “good” means lives in your head, not in the traces — if a tool could fully fix your product, it could fix everyone’s, and there’d be nothing left to differentiate yours AI’s real job: help you express and apply your judgment faster, not replace it The eval lifecycle (analyze → measure → improve) Error analysis — the hardest step: take traces and find failure modes. No perfect definition of “mistake” (you can’t define slop, but you know it when you see it) Measure — how prevalent is each failure mode? Pareto applies: ~80% of issues come from ~20% of failure modes — prioritize those Improve — fix the product: prompt instructions, model switch, fine-tuning. Iterate forever AI is weak at the front (taste-specific error analysis) and strong at the back (measurement, prompt optimization, hill-climbing). ...

Hamel Husain, a decade-plus AI consultant himself, argues the hottest job in the economy is usually a bad hire — with two exceptions. ~3 minutes on his own channel. Why hiring an AI consultant usually fails The problems you had before — talent, organizational, skill issues — are still there when the consultant leaves AI’s whole point is to remove the middleman: you express what you want in natural language and inject your domain knowledge directly into the tool If they just implement and walk away, you can’t maintain what they built — worst case you keep paying them, i.e. prompting a consultant instead of prompting an AI His track record: in over a decade, third-party AI implementations he’s seen don’t work When a consultant is worth it Education: the consultant up-levels your entire team and makes themselves obsolete — you keep the skills after they’re gone Skin in the game: they don’t get paid unless they affect your bottom line (increase revenue or decrease costs), demonstrated at a quality bar — aligned incentives The alternative Learn AI yourself — it’s the most accessible technology ever created; you mostly need to learn how to talk to it Coding skills help but aren’t required for lots of things; a growth mindset makes effective use tractable with practice “Instead of prompting an AI, you don’t want to be prompting a consultant.” ...

Hamel Husain answers a client question — how to eval an AI PE curriculum planner for K-12 — with a one-minute twist: evals and product trust are the same problem. ~1 minute on his own channel. The eval problem A client building an AI curriculum planner for K-12 physical education couldn’t figure out how to eval it The answer: think creatively — put yourself in the educator’s shoes Questions an educator actually asks Was this plan created by an expert? What other teachers are using this plan? Is this plan based on best practices and expert knowledge? The takeaway Infuse those signals into the product — they give the end user trust in what they’re seeing That same design doubles as your eval: provide users a way to assess what they see, and you can assess it too If you can’t eval your product, your product probably sucks “If you can’t eval your product, your product probably sucks.” ...

Hamel Husain on his own channel — a 1-minute clip on the “evals take too long” objection and why your users eval your product whether you do or not. The “evals take too long” objection One of the main reasons people skip evals: reading a trace takes forever — a lot of human hours you “don’t have time to do” His example: an AI data agent that answers business questions (supply, revenue) — “the most popular internal tool I’ve seen so far” The question: how do you eval that? If you can’t eval it, it probably sucks One principle to keep in mind: if you can’t eval your product, your product probably sucks — because your users need to eval the product When you present an AI answer to a user — say, an answer to a business question — you need to figure out a way to convey trust “If you can’t eval your product, your product probably sucks. Why is that? It’s because your users need to eval the product.” — Hamel Husain ...

Hamel Husain on his own channel — a 1.5-minute clip on LLM-as-a-judge evals: how to measure the judge’s noise and why a judge you can trust 100% of the time doesn’t exist. Measure the judge against ground truth An LLM judge is a “soft check” — you can scientifically measure how good it is by comparing it to labels Assemble a dataset of human labels as ground truth (is this thing good or bad?) and measure how much noise the judge has Even ~100 labels is enough to measure the noise There are different kinds of noise — you can tune the judge to reduce it, but there will always be some A judge is a black-box classifier There’s no such thing as a judge you can blindly trust with no noise at all It’s a classifier, like a stop-sign detector: get it very reliable and there’s still an edge case somewhere producing false positives and false negatives You have to be okay with that Tolerable noise is a business decision Tune the judge to reduce noise, then make the call: is this level of noise tolerable for your use case? Bake the eval’s known characteristics into your decision making — that’s the best you can do with an LLM judge “Even if you can get that to be very reliable, there’ll always be an edge case somewhere where it’s going to give you a false positive, false negative.” — Hamel Husain ...

Hamel Husain on his own channel — a 2-minute clip on why traditional scoring metrics (ROUGE/BLEU) and rating scales make evals harder to act on, and why he defaults to binary pass/fail. ROUGE/BLEU: string similarity from a different era These scores come from traditional ML/NLP — at their core they measure string similarity A coarse metric that made sense when LLMs could barely produce coherent language; “we’re way past the point of producing coherent English” now Part of the “eval industrial complex”: we have metrics, trust us because we’re experts — it feels fancy but isn’t what you need Rating scales hide uncertainty Unless you have a very sophisticated setup and put real resources into aligning scores with humans — which ~99% of teams don’t — a 1-to-10 or 1-to-5 scale is a bad idea Nobody knows what 4.2 versus 3.7 means Humans just hide their uncertainty in the middle values, so you don’t get good decision making out of it Binary pass/fail is what you can ship on At the end of the day you have to ship your product: is this good or not? Binary evals are easier to align with humans — which you always have to do with an eval — and easier to action A score of 3.2 → “I don’t know what’s wrong.” A fail → “okay, it failed, now you can action on that” “You see a score of 3.2, you’re like, I don’t know what’s wrong… but fail, it’s like, okay, it failed, now you can action on that.” — Hamel Husain ...

Hamel Husain on his own channel — a 40-second clip on why rating-scale evals (1–5) are the wrong default: he pushes for binary pass/fail evals with failures scoped into specific, actionable criteria. Binary beats a 1–5 scale He tries “really hard” to make every eval binary — pass or fail Scope the failure: “agent took too long → failed because it took too long”, “too many steps → failed because it took too many steps” He has never gotten stuck converting a 1–5 scale into binary, across all the companies he’s worked with The exception: a well-tested rubric everyone is confident in — then a 1–5 scale might genuinely work In most cases the scale just “kicks the can down the road” — the noise doesn’t disappear, it gets hidden “In a lot of cases, it just kind of kicks the can down the road, and you’re just hiding a lot of noise in there.” — Hamel Husain ...

Hamel Husain (founder of Parlance Labs, co-author of an upcoming book on AI evals with Shreya Shankar) on the Applied Intelligence podcast — 39 minutes on why evals are the testing framework most businesses skip, and how to think about them the right way. What an eval actually is An eval is structured data analysis and debugging on your AI application — you know what’s broken, and you know what to prioritize fixing It’s the answer to a real problem: AI outputs are stochastic (text), with no deterministic “right or wrong” to test against Software testing assumes a finite surface of failure; AI has an infinite one, so you have to decide what to measure Generic metrics are a trap Search “how to evaluate my AI app” and you’ll find vendors promising a dashboard of helpfulness, conciseness, toxicity, coherence scores (usually 1–5) Nobody knows what those numbers mean, they don’t correlate with what matters, and they actively burn engineering cycles looking at metrics that don’t matter “Reduce hallucination, increase helpfulness” is the same generic-metrics mindset — you’re going through motions without knowing what’s actually wrong There’s no easy button; you can use coding agents to write the evals, but you still have to be thoughtful about what you measure Start bottom-up, not top-down The process starts with error analysis: a blend of qualitative and quantitative work to find what’s broken in your application Most people are too top-down (worried about failure types they imagine) and get lost in generic metrics; the bottoms-up read of your actual data is the missing half Evals are really a process of eliciting your specification, then measuring against it — you can’t know “good” until you look at outputs and iterate AI can’t do your evals for you AI is great at fixing deterministic bugs; it cannot read your mind — it doesn’t know what “good” feels like for your product The future is AI walking you through the evals process and interrogating you, not doing it alone — a human stays in the loop Eval tooling needs to render your data domain-specifically (images as images, emails as emails, chat as chat), so it’s more than a chatbot Two failure modes Hamel sees most Not using AI deeply yourself — no coding with AI, no building with it → bad intuition, bad specifications Reaching for complexity too fast — day-one orchestration frameworks, graph databases, multi-agent setups before you can reason about what’s happening Deliberate, not slow AI lets you build the wrong thing faster — and many wrong things faster; it also lets you build the right thing faster You need product sense, grounding, and taste, or you just churn through bad ideas at higher speed AI amplifies who you are: if you’re okay with slop, it removes the friction and amplifies the slop Chatbot vs MCP Slapping a chatbot on an existing product is the mediocre default when there’s a “we need AI” mandate from the top Better: expose an MCP or an API on your product — the Google Workspace CLI is the canonical example of how much more useful agent-friendly access is Interface matters: a scheduling flow shouldn’t be a brittle text back-and-forth when a picker widget gives visual confirmation and avoids bugs Guardrails A guardrail is a specific eval sitting in the request/response path that blocks a bad output (competitor talk, profanity) Off-the-shelf guardrails are just someone else’s prompt, tuned to a different domain (shopping, travel) — you still have to do the evals work to know which failures to guard against Prioritize failures you can simulate or actually observe; unobservable ones are lower priority The stack, the vendor, and the team Start with the most powerful model you already know, build an eval harness around the metrics that matter, then back off to smaller/cheaper models and reason about the latency–cost–quality tradeoffs Data science gets more important, not less: the ability to ask the right questions is directly proportional to the quality of output you get AI competency is core to any knowledge-work business, so be careful outsourcing it — use third parties to upskill your team (a deliberate training exercise), never as a crutch The takeaway Parlance’s model is a “driving school” boot camp: they pair-program the whole end-to-end evals process on your data until you don’t need them The million-lines-of-code demos are all backed by harness engineering — metrics, logs, traces, observability — and evals are almost all of that harness You don’t need an R&D budget, maybe just a token budget: a $100 plan and deliberate experimentation gets you a long way “AI cannot read your mind. It doesn’t know what you feel like good is.” ...

An 85-minute conversation between Dan Shipper (Every) and Paul Ford — co-founder of Abort/Aboard, former Harper’s editor, author of the Bloomberg classic “What Is Code?” — recorded days after Opus 4.5 landed inside Claude Code, at a moment both describe as “the world changed last week.” Two deeply reflective practitioners trying to metabolize a genuine step change in real time. The step change Opus 4.5 is the first time vibe coding “just keeps going without tripping over itself” — it builds, and fixes its own errors Paul built a fully-featured iPhone reading app (photo → analysis → research agent → custom reading profile) with no idea how it works Paul’s framing: NOT a 9,000x model jump — a product step change. Claude Code added agent-style self-evaluation. It’s “the first true product built on top of an LLM” Claude Code’s design principle Anything you can do on your computer, Claude Code can do — low-level tools (files, grep, bash) below the level of features Features are just prompts: slash commands and subagents, writable in English General product principle: move what used to be code functionality into prompts the agent executes with low-level tools The emerging skill: abstraction-level thinking Don’t hand the agent the problem — hand it the way to get information about the problem, then constraints Paul’s synth pipeline: spider DSP textbooks into a SQLite reference → constrain to good open-source libraries → implement. Five or six levels up, and “make me a synth like this” works “That’s the skill that’s going to be emerging” The hard truths “I no longer feel I can in good faith say human skills are going to be relevant” — 600K jobs at Accenture alone, 50M devs worldwide “Everyone gets the same Pokemon shoved into the mailbox” — the power is universal, instantly The GLP-1 analogy: rules of a lifetime can change overnight, and a year or two is nowhere near enough to metabolize it “Software was eating the world. Now it’s eating itself.” His concept: latent software — the PDFs and spreadsheets that describe software that doesn’t exist yet The discourse taxonomy AGI-is-coming group: gone quiet because there’s money to be made. Sam Altman “wants to be Steve Jobs but he’s Steve Ballmer.” OpenAI is Microsoft; Anthropic is Google. Nobody is Apple — “you can’t put a civilian in front of that interface” Left-adjacent literary types (his Harper’s world): want their prose untouched Rejecters vs. do-gooders: charities and climate scientists can’t wait to use it to accelerate missions that are “unalloyed good” Professors who keep it away from students: he respects that line completely The real harms Provenance: “I want nutritional guidelines for what’s in my Anthropic LLM” — Google honors robots.txt; LLMs don’t tell you what’s in them The devaluation of the 50M-person underpinning of the global economy — “who gets to talk about that?” The failure to plan: “people see it coming but don’t really plan for it” The Sankey chart Paul had Claude build a mild-bearish model of consulting’s future: McKinsey $16B → $4B by 2035; Alexander makes partner in 2029 “just as the firm started its long contraction. She was one of the last… the smartest thing in every room now was the computer.” Shared with a consultant: “they got quiet for a minute. And they went, ‘interesting.’” ...

Horace He (Thinking Machines Lab) on one of the most annoying facts about LLMs: ask the same model the same question twice and you can get different answers — even at temperature 0, where the math says the model should always pick the same token. The common explanation is wrong The usual suspect: “concurrency + floating point” — GPU threads racing to accumulate results in different orders, so the same kernel gives different answers run to run But the same matrix multiplication on the same data is bitwise identical 1000 times in a row. GPUs are definitely concurrent and floating point is definitely involved — so that’s not the whole story The real foundation is floating-point non-associativity: (a+b)+c ≠ a+(b+c). Adding numbers in a different order genuinely changes the result. The question is what changes the order The actual culprit: your batch size Four things are simultaneously true: some GPU kernels are nondeterministic; every kernel in an LLM’s forward pass is deterministic; the inference server is deterministic; and yet users see nondeterminism. ...