Hamel-Husain

Hamel Husain walks engineers and product managers through AI evaluation take-homes using a hypothetical tree-care app (21:18, including Q&A). Start with the data, not the coding agent Spend 50–70% of the time understanding the data, Hamel recommends; inspect actual examples rather than asking an agent to discover every error. His fictional Tree Buddy app combines input checks, species identification, data extraction, and care recommendations—each needs a different evaluation. Find missing labels, inconsistent units, ambiguous categories, and examples that do not match the product’s intended use. A guardrail dataset containing only allowed inputs can yield 100% accuracy while testing nothing about rejecting bad inputs. Use trustworthy external data where appropriate; synthetic negative examples can fill gaps, but remain proxies rather than proof of real-world coverage. Run experiments you can explain Begin with a simple prompt-only baseline, then compare targeted changes such as retrieval, a different model, or prompt optimization. For each feature, record the goal, method, results, and what the comparison taught you. Keep development examples separate from held-out test examples so repeated tuning does not masquerade as general improvement. Compare quality with speed, cost, and complexity; the highest score need not produce the best product. The model names, dataset counts, and result rows in this example are illustrative—not benchmark findings. Challenge weak metrics constructively If asked to combine one-to-five helpfulness and tone ratings, provide the requested calculation and explain its limitations. Propose concrete checks: is the photo sufficient, is watering advice grounded in evidence, and does the response avoid blaming the user? If an AI grader cannot be checked against human labels, explicitly say its scores remain unvalidated. Explain problems politely: a weak assignment may be a deliberate test of judgment, or simply a poorly designed interview. Make the submission easy to review Structure it as executive summary → data → experiments → results → learnings and limitations → next steps → AI-tool disclosure. Keep the executive summary to roughly a third of a page; synthesize the winning approach rather than repeating every experiment. Name the exact additional data and product changes you would pursue—not just “get better data.” Follow the requested PDF or repository format; a flashy generated UI is not a substitute for a readable report. PMs should demonstrate product judgment and data literacy; coding agents and eval skills are useful only when the candidate keeps thinking. “Never assume a data set you are given in real life and especially in an interview is quote correct.” ...

Hamel Husain and Isaac Flath test Anthropic’s automated evaluation-building skill on real apartment-leasing conversations. A 48-minute live walkthrough, not a controlled benchmark. What Claude gets right It identifies genuine transfer and scheduling failures in a dataset Hamel already knows well. It notices voice and text have different instructions and retracts some suspected pricing errors. Its advertised principles are sensible: realistic tasks, room to improve, stable grading and human review. Where the workflow goes wrong It recommends a test before the humans have inspected the conversations or compared user impact. It asks reviewers to check model-generated labels in Markdown, with truncated tool results. It initially presents aggregate scores rather than the individual evidence needed to challenge them. A usable review page appears only after Hamel explicitly asks for one; annotation persistence remains fragile. The grader tests the wrong things Literal confirmation wording is brittle: callers can request a human without using a particular phrase. Repeated questions, failed transfers and awkward narration are different problems, not one pass/fail bundle. A bot promising a transfer without making the tool call is more consequential than a verbose confirmation. Agreement on “fail” is not enough if the grader gives the wrong reason. The better starting point Build a readable viewer that preserves messages, instructions, tool calls and tool results. Read real conversations and write your own observations before seeing suggested verdicts. Prioritize failures by frequency and user harm; fix obvious bugs without automatically creating a new eval. Build focused tests, validate their case-level reasoning, then optimize against scores you trust. Both speakers reject this workflow in the demonstrated run, while acknowledging better failure discovery than the earlier tool they tested. They do not demonstrate the separate hill-climbing stage, and they are testing recorded traces without application code. ...

Hamel Husain distills recurring mistakes from teaching AI evaluation to more than 5,000 engineers and product managers and working with over 50 companies (12:47; figures are his own). Start with observed failures 1. Writing tests before reading data: inspect real interactions before brainstorming an unlimited list of possible problems. 2. Using generic metrics: replace abstract helpfulness scores with specific failures such as missed handoffs, incorrect recipients, or broken scheduling. His leasing example: a polite goodbye looked successful but missed the opportunity to offer a cheaper qualifying property. Favor non-trivial, observed or reproducible failures and verifiable pass/fail checks—not giant catch-all prompts. Validate the evaluator and design for review 3. Trusting an untested AI judge: compare its decisions with domain-expert labels before trusting the score. His suggested split is 20% examples, 40% development, and 40% held-out testing; tune on development, then check unseen cases. Examine which failures the judge misses and which acceptable results it wrongly flags, not just overall accuracy. 4. Building opaque products: show sources, intermediate calculations, and relevant comparisons so experts can inspect an answer efficiently. For lesson plans, show changes against a trusted plan and let teachers accept or reject them. Keep criteria simple and humans accountable 5. Using one-to-five ratings: narrowly scoped pass/fail judgments are easier to align with a “good enough to ship” decision. 6. Asking the wrong people to review: lawyers judge legal assistance; medical experts judge medical assistance; developers are not automatically domain experts. Build a custom annotation interface that makes the expert’s review easy. 7. Outsourcing discovery to automation: automated tools can catch obvious defects while missing business context and interaction failures. In his vendor comparison, tools missed leasing objections, markdown in texts, voice interruptions, and human handoffs; this is a reported study, not a universal ranking. “It’s impossible for you to know what good looks like without looking at some data.” ...

Ben Hylak of Raindrop joins Hamel Husain for a 40-minute workshop on testing agent changes before they reach users. Test the whole agent, not just its prompt Tools, field names, memory, subagents, and external data can change behavior independently of the main prompt. A trace—the record of an agent’s messages, tool calls, and results—captures more than a single input/output test. Production monitoring finds unexpected failures, but cannot prevent the first bad outcome. Replay the cases your change touches If you remove a tool, start with historical traces that actually used it. Re-run the changed agent and inspect differences in answers, tool errors, step count, cost, and latency. Hylak suggests roughly 5–10 targeted scenarios for a small change, and perhaps 50 or more for a harness rewrite—not universal coverage guarantees. Mix known failures with representative cases; repeat runs to account for nondeterministic behavior. Reconstruct the world around the agent Live databases have moved on: yesterday’s delivery order cannot be tested against today’s order status. Cached tool responses break when the new agent asks different questions or takes a different path. Preserve real code and validators where possible, then simulate external responses at the network boundary. Historical tool results are partial windows into the old world; use them to build a coherent, stateful test environment. Predicting a human’s next reaction is a separate, much harder problem—not what this replay approach solves. Check that the simulation is believable Empty sandboxes, inconsistent dates, and mismatched responses can make an agent recognize the test and behave differently. First replay the unchanged production agent: its simulated behavior should align with the original traces. A blind model comparison of real and simulated traces can expose artifacts, but is not proof of complete fidelity. New data sources may need supplied fixtures or controlled read-only access during environment construction. Turn traces into focused checks Separate diffs (what changed?) from evals (does this meet our definition of acceptable?). Start with cheap checks: tool errors, call counts, output length, cost, and latency. Add model-based judgments where meaning matters; define the criterion narrowly and prefer clear pass/fail decisions. Replay bad traces to ask whether the original failure still occurs; use good traces to protect working behavior. Surface changed or unusual traces for human review instead of drowning reviewers in generated summaries. Start small; keep production as the final check Run the agent locally and inspect every tool call before buying elaborate infrastructure. Build a simple simulator when the tools and data sources are manageable. Shadowing all traffic can be expensive and slow; targeted simulations tighten the feedback loop. Raindrop’s implementation and efficiency claims come from its co-founder, not an independent benchmark. “What will my change change?” — Ben Hylak ...

Hamel Husain’s 38-second clip challenges the reflex to turn an executive AI mandate into a chatbot on an existing product. YouTube disables embedding for this video; the header links directly to it. Start with the user’s work A bolt-on chat window can check the “we use AI” box without making the product meaningfully better. Ask which existing task could become faster or easier for the user. Depending on the workflow, exposing an API or an MCP interface—a way for AI tools to interact with the product—might be more useful than adding chat UI. He offers alternatives, not evidence that either interface is always the right design; the short does not show a worked example. “Be thoughtful of how you can actually help the user … accelerate their workflow.” — Hamel Husain ...

Hamel Husain and Isaac Flath test a claim about Opus 5.5’s writing in a 20-minute blind, side-by-side review against Opus 5. Their verdict is deliberately narrower than “slop is solved.” How they compare A small app hides the model names until review; the speakers use the same writing tasks and inspect outputs aloud. Tasks include an AI-evals FAQ, a skeptical-reader summary of Isaac’s Jev article, replies to technical posts, an intentionally cringeworthy influencer post, and a conference abstract. They judge flow and reader context, not just grammar or whether a response includes all the facts; they notice when one model makes fewer tool calls but cannot tell whether it read the same context. What improves—and what doesn’t In the Jev summary, 5.5 opens more naturally, then swerves into numbers and task details before explaining why they matter; 5 is awkward and needlessly defensive. Replies such as “Great write-up” still sound like engagement bots. Prompting for an influencer voice elicits the expected hooks and clichés from both models. A request for a short, thoughtful reply gets outputs they reject from both models; the conference abstract has slightly clearer framing from 5.5 but remains generic and needs heavy editing. Across this informal handful, 5.5 tends to fare better, but both use strikingly similar structure. This is not a rigorous benchmark or evidence of why the model changed. A more useful writing eval Ask whether an unfamiliar reader knows what a number or example refers to before it appears. Prefer specific, grounded examples to an inventory of details dumped into an introduction. Add a both failed option rather than forcing a winner between two drafts you would rewrite. Blind the labels and inspect the entire draft: an opening can sound good while later sentences fall apart. “Slop is not dead. 5.5 might be a little bit better in some cases, but it’s not like a landslide.” — Hamel Husain ...

Hamel Husain explains how to inspect AI-product failures when sensitive customer information restricts access, in an 81-second clip on his channel. YouTube reports embedding disabled for this video; use the watch link above. Work down the access ladder Get permission for real samples: Ask for approved test users, test datasets, or a limited user subset. Partial authorized access beats no inspection. Partner with authorized experts: If only doctors or lawyers may inspect the traces, make the product easy for them to verify rather than routing sensitive data to unauthorized developers. Redact traces: Remove sensitive details before reviewing them. Whether a library or LLM does the redaction, rigorously test that private information actually disappears. Use synthetic data only as a last resort: It may reveal obvious problems, but is unlikely to expose the subtle errors that occur with real users. “There is no replacement for looking at real data.” ...

Hamel Husain ranks common AI-evaluation practices after working with more than 50 product teams, in a 13-minute video on his channel. Start with real work, not a scorecard S: Error analysis and reading actual agent traces — inspect failures before deciding what to measure. S: Build an annotation view tailored to the product: render an email-writing agent’s output like an email, not a spreadsheet row. C: Brainstorming hypothetical failures or writing tests before building can start a conversation, but is a poor way to prioritize an unlimited failure surface. F: Generic helpfulness/coherence scores, vendor-provided judges, or choosing an eval vendor before understanding your own data. Build checks people can trust A: Binary pass/fail criteria beat vague 1–5 ratings; middle scores obscure the decision you actually need to make. S: Compare LLM-judge decisions with human labels. Prefer cheap code-based checks where they work, and validate those against people too. F: One judge prompt for every kind of failure, or off-the-shelf guardrails not tuned to the product. D: Letting only developers annotate a medical or legal assistant; the relevant domain expert needs to judge the outputs. S: Put domain experts close enough to the product to write prompts, not merely review them at the end. Manage the evaluation portfolio A suite that always passes (D) isn’t stretching the product. Run persistently passing, costly checks less often (A); not every isolated failure warrants its own eval (F). Public benchmarks may help compare models, but cannot replace product-specific checks (F). A trusted decision-maker for each feature or domain can break annotation deadlocks (A); showing intermediate outputs makes expert verification easier (B). Prompt optimization without evals (F) and letting AI do all the evaluating (F) cannot substitute for people inspecting difficult, contextual failures. “Look at data first.” ...

Hamel Husain explains his advice for reviewing AI-agent traces—records of what an agent actually did—in a 38-second clip. YouTube reports this upload cannot be embedded; use the Watch link above. Why 100? “Read 100” gives teams a concrete goal that gets them past the anxiety of starting an open-ended review. It is not a statistically proven threshold or a magic number. Without a target, “look at the data until you know when to stop” is too vague to motivate action. Once people approach the target, the guidance becomes more nuanced: keep reading until new traces stop revealing new patterns. Hamel calls that theoretical saturation. “100 is not a magic number. It’s just a cognitive unblock.” ...

Hamel Husain answers a question about stale gold datasets in a one-minute clip. YouTube reports this particular video cannot be embedded; use the Watch link above. Keep the challenge relevant Gold datasets go stale as a product improves; periodically inspect new errors and add fresh examples. LLM-judged evaluations cost money to build and run. An eval that always passes may no longer be worth running as often. Evals are a moving target for improvement, not a fixed scoreboard to optimize forever. For a stable long-term comparison, watch product metrics such as churn, revenue, and monthly active users instead. “You should always have a new challenge as you get better.” ...

Isaac Flath walks Hamel Husain through a 42-minute hands-on evaluation of OCR and document-question-answering agents. His key move: inspect the document, extracted text, retrieval, citations, and agent trace together. Why a wrong answer may not be a model failure A misread digit in a financial table can look like an LLM hallucination; better prompting cannot correct corrupted input. Documents mix scanned images, text layers, tables, watermarks, and handwriting. Medical, legal, and financial workflows can turn small extraction errors into consequential answers. Judge errors by the task: a shifted underline may be cosmetic; a missing inspection score or draft watermark may change the meaning. Inspect real documents, not just benchmark scores On an ordinary food-inspection form, a heading spans rows inconsistently and an OCR model drops a point value. Even the presenters debate the correct grouping. Compare the PDF, raw OCR, and rendered output side by side. Attach each observation to both a page bounding box and the corresponding extracted-text span. A different model invents checkboxes, converting ambiguous fields into binary ones; both OCR systems miss an “example” watermark in a medical record. A missed “draft,” “confidential,” or “void” stamp could be much worse. Flath built his annotation UI incrementally, starting in about an hour and adding rendered views, boxes, and hover-linked text whenever manual review became painful. Trace an agent failure backwards Check the final answer and whether its citation actually supports it. Inspect the agent trace: was the wrong number already in its context? Check retrieval: did it fetch the wrong page or the right page with wrong text? Compare extracted text with the original PDF. If OCR broke it, test and fix that layer, not the prompt. Add a cheap targeted check when possible, such as verifying table entries add up to the stated total. Evals across the pipeline Test OCR, retrieval, and the agent loop separately; swapping an agent harness will not repair a faulty source representation. A citation’s presence is weaker than checking whether the cited image/text supports the answer. Visual, page-level citations make human verification practical. Tool-using agents can crop and re-OCR pages until they find an answer, but sometimes take 45 minutes and spend $20, even when the answer is absent. “Blank” versus “zero” in a loan estimate, or a faint signature on a contract, requires domain judgment; not every discrepancy admits an automatic yes/no label. “Maybe 80% of my failures are actually just because my OCR model failed, in which case prompt tuning is just a waste of my time right now.” — Isaac Flath (hypothetical example, not a measured result) ...

Hamel Husain explains why annotation work should usually begin with open-ended observation rather than a premature rubric or “correct” reference answer. Start without the answer key A reference answer is an example of a correct response. A rubric is a set of criteria used to judge a response. Neither is required before inspecting data; defining them too early can bias annotators toward the team’s initial assumptions. Husain calls the discovery of better criteria while examining real examples criteria drift. Let traces reveal the problem Imagine a support agent that follows the written refund policy and escalates correctly. Reviewing traces may reveal that the escalation itself is clumsy and frustrating, even though it satisfies the original specification. An early rubric focused only on policy compliance would miss that product failure. The sequence that works Review representative traces without forcing observations into preset categories. Write open-ended notes about anything confusing, incorrect, or harmful. Group similar notes and count them to expose recurring patterns. Turn the observed patterns into a stronger rubric and better reference examples. “It’s often better to wait until after you look at some data before you specify a rubric or have reference examples.” — Hamel Husain ...

Hamel Husain and Isaac Flath examine DataLab’s accessible-PDF pipeline and explain why plain OCR output is inadequate for screen readers—and often for downstream AI systems too. What accessibility requires A reader using a screen reader, Braille display, or reflowed text needs the same context, sequence, and navigation available to a sighted reader. PDFs often expose no text layer, a broken one, or text in the wrong order across columns, tables, and page breaks. Visual cues such as headings, indentation, lists, captions, and page furniture carry structure that a flat text dump loses. Structure, not just characters The pipeline identifies headings, paragraphs, lists, formulas, figures, form fields, and the intended reading order. Figures are cropped and described; equations are represented structurally so assistive software can handle them differently from prose. A screen reader can announce content types instead of reading a math expression as disconnected symbols. The same structured output can help an LLM distinguish a table, footer, list, or formula instead of receiving one garbled string. Why the economics matter Manually remediating a complex PDF can cost more than $100 because someone must reconstruct its semantics and reading order. DataLab says its API reduces that first pass to pennies, but it does not claim automatic 100% compliance. A Utah educator producing accessible STEM materials reportedly reduced work from hours to minutes. High-stakes accessibility still needs human review; automation changes the amount of labor rather than eliminating responsibility. The broader lesson for document AI OCR quality is not only character accuracy—the output must preserve how the document is meant to be understood. Different downstream uses may need different formats, including tagged structure, HTML, JSON, or LaTeX-like math. Models are effectively “blind” when document-processing pipelines throw away layout and semantics before inference. “Just getting text and jamming it all together loses a lot of that structure.” — Isaac Flath ...

Hamel Husain uses a 39-second example to show how a high agreement score can hide a judge that detects no failures at all. The accuracy trap Suppose a product fails on 10% of examples and passes on the other 90%. A judge that always predicts “pass” will agree with the human labels 90% of the time. The headline score sounds strong even though the judge misses every failure—the cases the eval was built to find. What teams should inspect Treat overall agreement as a warning sign when one outcome is much more common than the other. Break results into false positives and false negatives instead of compressing them into one percentage. Check recall on the failure class: of the real failures, how many did the judge actually catch? Review the confusion matrix and the underlying labeled examples before presenting the metric to stakeholders. The product lesson A metric is useful only if it measures the behavior that matters operationally. Product managers should pause whenever “agreement” appears without a class-by-class error analysis. A judge that reproduces the majority label is not necessarily evaluating anything. “You can agree with it 90% of the time by just saying it never fails.” — Hamel Husain ...

Hamel Husain draws a one-minute boundary between two things that are both called “evals” but answer different questions. Model benchmarks Tests such as SWE-bench and GPQA Diamond compare general model capability across coding or advanced academic tasks. They help with model selection and track broad progress. A strong score does not establish that a complete product will perform its own workflow correctly. Product evals A product eval measures the behavior of a specific application—including its prompts, tools, retrieval, orchestration, and business rules. The test should express what success means for the actual user task. For an order-management agent, that might mean checking both that the correct order was selected and that the cancellation really occurred. The practical distinction Use benchmarks to learn what a model can do in general. Use product evals to learn whether the system you built does what you intended. Shipping decisions require the second kind, because users interact with the product—not an isolated leaderboard score. “Product evals measure whether your specific AI product does what you want it to do.” ...

Hamel Husain and Isaac Flath try a real delegated purchase: a Mercury agent card, Codex, a $5 course, and a checkout flow that turns into an eight-minute tour of the remaining friction. The setup Mercury’s agent card exposes credentials through its API and CLI so an agent can enter them at checkout. The owner chooses the spending limit and explicitly grants access to that card. The agent cannot create another card, raise its own limit, or reach cards it was not given. Isaac compares the design to an API key with scoped permissions; Hamel caps this card at $100. The task Find Isaac’s course and Mercury’s agent-card documentation. Apply a coupon that lowers the course price to $5. Use the already authenticated Mercury CLI to retrieve payment details. Confirm before buying and avoid exposing the card on the live stream. What went wrong Codex tried incorrect versions of the coupon code, then appended text instead of clearing the field. It needed Hamel’s existing signed-in browser profile rather than completing the flow cleanly in its own browser. It stopped before entering full card credentials because authorization to spend $5 was not interpreted as authorization to disclose those credentials to Stripe. Billing-address handling required another manual intervention. The final payment triggered repeated risk denials even after Hamel approved the transaction. What the friction revealed The guardrails did real work: a compromised or confused agent could not silently reach an unlimited account. But the system distinguished intent, credential disclosure, billing data, and final submission as separate approval boundaries. For a one-off $5 purchase, Hamel could have completed the checkout much faster himself. The workflow becomes more plausible for asynchronous delegation—while walking or driving—where elapsed time matters less than avoiding direct interaction. Repeated low-risk purchases might justify a reusable skill and a preapproved threshold, but “always authorized” would remove the protection the scoped card is meant to provide. The result After several corrections and approvals, the charge reached Isaac’s Stripe account. The purchase succeeded, but neither participant considered the experience smooth enough for routine use. The experiment is a useful reminder that financial agents need both a small blast radius and a usable authorization model; either one without the other fails. “We have purchased something with an agent. My first time. … But it was painful.” ...

Hamel Husain walks through eight agent skills he and Shreya Shankar built from work with 50+ companies and 5,000+ eval-course students. The 12-minute tour focuses on a simple correction: inspect real failures before inventing tests. Start with the router eval-start is the entry point — it inspects what you already have and routes Claude Code or Codex to the right specialist skill Existing traces but no failure taxonomy → start with error discovery Existing evals that may be unreliable → start with the eval audit Hamel’s priority: master those two before reaching for the rest of the toolkit Error discovery: turn traces into a failure taxonomy Understand the data — parse its structure, identify the dimensions that vary most, and ask what “bad” could mean Design a review interface — use color, spacing, contrast, typography, and containers to make the important differences easy to scan Build the interface around the data — a multi-turn voice-agent trace needs a different view from a document or search result Sample for discovery — combine random examples with cluster-based diversity instead of reading logs one by one Adapt while the human reviews — annotations steer the next sample toward both breadth across the dataset and depth around a newly found failure The goal at this stage is finding kinds of failures, not estimating how frequently each one occurs The loop keeps serving examples until the reviewer stops discovering new failure modes The human supplies judgment; the coding agent handles parsing, interface construction, clustering, and the repetitive search for related traces The eval audit: catch common methodology mistakes Confirms that proposed checks come from observed, annotated failures rather than brainstormed hypotheticals Prefers focused binary pass/fail criteria over noisy letter grades or Likert-style scales Flags over-scoped LLM judges and recommends deterministic code checks where code can answer the question Warns against treating text-similarity scores as general-purpose quality measures Requires each LLM judge to be tested against human labels Checks train/dev separation so judge prompts are not tuned on the same labels used to report their quality Reviews annotation practice, evaluator maintenance, recurring error analysis, and coverage per failure mode; Hamel offers roughly 100 labeled traces per failure mode as a useful target, not a law Produces a diagnostic report instead of silently changing the eval setup The other six skills Generate synthetic data — stress-test a pre-product system when real users and traces do not yet exist, with emphasis on diverse edge cases Write a judge prompt — create a narrowly scoped LLM-as-judge rubric using patterns that held up across consulting engagements Validate an evaluator — compare a judge with human labels using clean splits before trusting its scores Evaluate RAG — build checks around retrieval and search quality, not only the final generated answer Build a review interface — generate a purpose-built annotation UI; error discovery calls this skill internally eval-start — tie the workflow together and route the agent based on the project’s current stage The practical order Start with traces from the actual product Use error discovery to understand the failures and create labels Audit the resulting eval plan for weak criteria, leakage, and unvalidated judges Only then add synthetic cases, judge prompts, retrieval metrics, and production automation “The goal is discovering failure modes, not estimating how common they are.” ...

Hamel Husain and Isaac Flath continue their live comparison of writing prompts with a five-minute clip about an inconvenient result: the carefully crafted prompt does not reliably beat the alternatives. The test Three setups answer the same writing task: no system prompt, Anthropic’s long mannered-prose guidance, and a shorter plain-writing prompt Anthropic’s guidance tells the model to prefer literal statements over metaphors such as “a dial worth turning” Hamel expected the guidance to backfire because it puts examples of the unwanted style directly into context The hosts compare the drafts by reading them, choosing between them, and discussing how much editing each would require What happened No candidate wins every example; useful passages and awkward “AI slop” appear across all three The long anti-slop guidance performs much better than Hamel expected, despite being written partly in the style it criticizes A short prompt can be competitive with the elaborate version, and sometimes no writing prompt at all is close enough to win The better output depends partly on the goal: the draft with the strongest structure may still need more line editing than a plainer alternative The practical lesson Treat writing prompts as candidates to evaluate, not permanent truths to adopt on reputation Compare them on several representative examples; five examples can reveal surprises but cannot establish a universal winner Judge the complete working context — source material, task, examples, and desired output — rather than crediting the system prompt alone Optimize for the draft that is easiest to turn into good writing, not the one backed by the most impressive prompt “There is no winner all the time necessarily. You could have good things from each.” ...

Hamel Husain on his own channel — a 57-second answer to the question every team doing evals eventually asks: how do you steer an agent to do this work well — skills, MCP servers, agent markdown files? Start with the person, not the agent The question: how do you constrain or steer the agent to do evals well as a human partner? Is it equipping it with skills and MCP servers you built and maintain, or agent markdown files? Hamel’s answer does not start with tooling: “my answer starts with the person, not the agent” You have to educate the human first, and the human has to understand the process — what good looks like, and what you are actually trying to do Only after that do the skills help; the agent is not the bottleneck the question assumes Why skills have a ceiling Skills “can only help you so much” — there is an upper limit to what they can do for you The reason is structural: a published skill is generic by design, because it has to apply across a lot of different use cases Getting value out of it means putting in “a little bit more thinking” to customize it for your own problem The whole thing “comes down to learning what good evals look like, plus using the skills, and then customizing the skills” for yourself The loop he describes Learn what good evals look like — the judgment, not the tooling Use the skills Customize them to your use case Same frame he applies to coding: the skill is scaffolding around the process knowledge, not a replacement for it The resources in the clip The evals skills are open — hamelsmu/evals-skills on GitHub, with updates landing as recently as the day before this clip The next AI Evals cohort with Hugo (Bowne-Anderson) is open at maven.com/parlance-labs/evals “It’s just like anything else, it’s just like coding or anything else. You need to educate the human and you need to give them some skills, and the human needs to understand the process of what good looks like. The skills can only help you so much.” — Hamel Husain ...

Hamel Husain and Isaac Flath run a model’s writing through the cheapest eval there is: read the drafts out loud and argue about them. (Two-minute clip from the same “Writing Evals With Isaac” live session.) The task Isaac’s blog post describes a document QA pipeline built over a 77-page set with 22 questions. Every answer cites a bounding box drawn on the exact spot on the page, so checking an answer takes seconds instead of a full re-read They hand the post to a model and ask for an X post about it, focused on the pieces of the pipeline that can be tested Three drafts, one task, different prompts — same three candidates as the session’s writing-style comparison What they liked A real number, early and concrete: the model answers 197 for the underwriting fee — and “if you have to reread the whole document to find out, the answer saved you nothing” The pipeline stated as testable stages with the reason for splitting it: once checking is cheap, experiments are cheap, so you can swap one stage at a time and rerun the same 22 questions One fewer fact. The winning drafts carried less than the others, and that was an improvement rather than a loss Precision in naming: “finding (grep vs semantic search), cheapest run” beats “GPT vs semantic search over the same OCR output, same accuracy, cheapest run” — the longer version buries the point that the bottleneck was never search What they didn’t “Very similar but a little bit more wordy” is the entire verdict on one draft Drafts that pull the pipeline’s three steps out of the source post without explaining what they are — accurate and unreadable An extra fact about cost that added nothing to the argument Editorial framing that tells the reader what the result means before showing it The rule the description writes down An AI draft can get the facts right and still not be good. The ones we liked had a real number in them and one less fact — turns out cutting helped more than adding. ...

Hamel Husain and Isaac Flath take Anthropic’s new anti-slop writing guidance and put it through an actual comparison: one question, three system prompts, and a verdict reached by reading the drafts out loud. (Seven-minute clip from their “Writing Evals With Isaac” live session.) The thing being tested Anthropic published a named anti-pattern for its newest writing model, Claude Fable 5.1: mannered prose — metaphor and flourish substituted for direct statement The prompting guide treats it as a model-specific behavior. Fable 5.1’s writing is a step up from earlier Claude models (fewer stock phrases, less unexplained jargon), but in some cases it runs denser than Fable 5 — longer sentences, fewer paragraph breaks, more figurative phrasing standing in for plain statement The fix is one paragraph, pasted into a system prompt: Mannered prose substitutes metaphor and flourish for direct statement. Instead of “a parameter worth varying,” the mannered writer produces “a dial worth turning.” Instead of “this point still matters,” they write “this point earns its keep.” The phrases exist to display the writer, not to convey the idea, and readers can tell. That is why mannered prose irritates: it makes the reader work harder so the writer can perform. It is also imprecise. Metaphors drag in connotations the writer did not choose and cannot control. The fix is to say what you mean. When a literal phrase is available, use it. ...

Hamel Husain and a co-host spend an hour live-testing claude plugin eval — the plugin-evaluation feature Anthropic shipped for Claude Code (announced from the Claude Devs X account) — by pointing it at a plain-writing/tone skill. They never get a useful eval out of it, and the diagnosis of why is the interesting part. ...

Nandan Thakur — the University of Waterloo PhD behind BEIR, MIRACL, FreshStack and TREC-RAG, now a postdoc at Microsoft Research India — on Hamel Husain’s channel, ~51 minutes on the three things you need if you’re building a search agent: a benchmark you can actually reproduce, synthetic training data you can afford, and a way to see what the agent is doing under the hood. He’s upfront that the field is young: “not many things are formalized yet.” ...

Hamel Husain interviews Teresa Torres — author of Continuous Discovery Habits and the coach who popularized the Opportunity Solution Tree — ~91 minutes on why most teams build the wrong thing, what “talk to your customers” actually requires, and how evals are the missing discovery habit. Hamel’s audience is engineers, and Teresa’s audience is product teams; the conversation is the overlap, ending in one of the best eval war stories Hamel says he’s heard. ...

Hamel Husain (Parlance Labs, evals course co-author with Shreya Shankar) on Vanishing Gradients with Hugo Bowne-Anderson — ~76 minutes on why verification should drive AI product design, and how evals changed once agents arrived. The whole conversation orbits his recent post, “It’s hard to eval is actually a product smell.” ...

Radu, a search engineer at Vespa, on how to actually choose and tune embedding models for search and RAG — hosted by Hamel Husain. ~22 minutes. Why not just take the leaderboard Most people pick embeddings one of two ways: the top of a benchmark leaderboard, or the model from whatever provider they already use — leaving a lot on the table Cost, latency, robustness to future model changes, and your actual use case aren’t reflected in a single ranking MTEB is a starting point, not an answer: find the subtask closest to your use case, restrict by model size, and watch for small models that punch above their weight (this talk covers dense embedders — one vector per chunk) What MTEB doesn’t tell you Quantization depends on where you run: on CPU, int8 keeps most of the precision at a fraction of the cost and runs much faster; on GPU, an int8 model runs at its native rate and is actually slower — run FP16 instead, basically the same result for significantly less Vector precision: storing vectors as bf16 instead of float32 is a no-brainer (no measurable loss); bit vectors (packed to int8) do cost quality — but in hybrid search the gap shrinks and they become viable Matryoshka dimensions: some models are trained so the earliest dimensions matter most, so you can just cut the vector — 2048 → 1024 dims cost nothing, 512 a little, 256 more; and 2048-dim bit vectors run a quarter the size of bf16 at 512 dims Query latency comes from two places: how fast the embedder turns the query into a vector, and the distance math — normalized vectors let you use dot product instead of cosine, bit vectors unlock Hamming distance (fastest, and CPU-optimized in modern search engines) Constraints MTEB never covers: multilingual needs, long-context support Tune on your own data with Vespa Embed Open-source fine-tuning tool (UI over sentence-transformers, models from Hugging Face): feed pairs (query → document) or triplets when you have negatives; it auto-splits a validation set MNR (multiple negatives ranking) treats other documents in the batch as negatives; symmetric and JSD variants check whether a batch-picked document is actually a negative Start with defaults; if you have labeled negatives, try triplets — his e-commerce results were similar either way, but real hard negatives (from search logs: positives rank on top, negatives below) are worth mining No clean labels? Use an LLM as a judge: have it rank query→document pairs on a 0/1/2 scale, give it examples (the most important part), and run micro-batches of 8-10 docs so it doesn’t assume listwise context Fine-tune embeddings far more readily than LLMs — it’s a constrained problem: cheap, quick, and consistently a big NDCG jump The cheap-embedder escape hatch: re-rankers Two-phase search: an intentionally “great but not greatest” cheap embedder does first-phase ranking over millions of docs, then a more expensive re-ranker (float vectors, cross-encoders, late interaction) scores only the top N Store cheap bit vectors (with HNSW) in memory for the first phase and keep float vectors on disk for the re-rank — memory is expensive, disks aren’t Late-interaction re-ranking (MaxSim) is literally: per-token dot products, keep the max per patch, sum across the document Do’s and don’ts from real experiments Don’t train on title + description as a proxy for queries — proxies lie; people’s real queries look nothing like product copy. Train on real queries and relevance judgments NDCG has a blind spot: change your relevance function and newly-surfaced documents score zero until they’re rated — if you don’t re-rate everything in the metric, your NDCG looks like it never improves Pick the metric by the problem: e-commerce ranking → NDCG/ERR (top results matter); RAG → precision matters more, because junk retrieved = hallucination, and recall is capped by context size Don’t over-optimize: with ~2M documents most of these knobs don’t matter yet — start with defaults that lose little, then measure “The embedding is a much more constrained problem — and it’s a lot easier to fine-tune. I always find a really good benefit from doing it. And it’s not that complicated.” — Hamel Husain ...

Adam — Modal engineer, maintainer of Prefect and FastMCP — on Hamel Husain’s channel, making the case that environment engineering, not agent design, is the hard part of background agents. ~27 minutes. The thesis Agents have converged on a universal design (harness + context); the differentiator is the environment they operate in An homage to Anthropic’s “Don’t Build Agents, Build Skills” — one layer up: “the hard part of background agents is the background part, not the agent part” Dev boxes, not sandboxes A bare sandbox can’t run a server, take screenshots, or profile on a GPU — an agent is doomed without a machine prepped for its task CI/CD treats environments as jobs: clone, run deterministic tests, throw away Agents need sessions: resumable state (snapshot the filesystem, resume mid-thought), an environment that’s alive (repo drifted, ffmpeg now needed), a different security boundary, and cold starts that don’t make you wait 2-10 minutes every spin-up What Ramp and others converged on Per-repo images defined in code, rebuilt on a 30-minute schedule so agents boot warm in under a second Secrets kept out of the environment — agents submit requests to a proxy sidecar that attaches credentials deterministically Result: ~1M sandboxes, ~70% of Ramp’s PRs from background agents Async image builds mean humans never pay for the expensive rebuild Control plane vs data plane The old pattern (agent + tools in one process) means one aberrant tool call kills the agent holding the state Put a fire door between the planner and the executor: the agent lives in the control plane and executes work in a throwaway dev box — at worst the data-plane environment gets corrupted, never the agent Same principle as Airflow never running Spark itself Build vs buy The dev box supply chain is where to invest; Modal’s docs/examples cover single coding agents, a Lovable-style builder, LangGraph, computer-use, code interpreters Observability of “software factories” — instrumenting the impact of hundreds of agents, not one agent — is an underexplored open problem “The hard part of the background agents is always the background part and not the agent part.” ...

Shreya Shankar (evals researcher, taught the evals course now at 4,500+ students) and Hamel Husain on Peter Yang’s channel — 54 minutes of live demo: using Claude Code to turn your taste into evals, plus Hamel’s benchmark of the “auto-eval” vendor tools. Evals still start with data The fundamentals haven’t changed: look at data first, do error analysis, externalize your taste and judgment before writing any eval What changed: agents are now good enough to help you look at the data — running in the background while you review, giving you leverage in that first stage They’ve become bigger fans of LLM judges: an LLM judging a trace against one very specific, well-defined criterion (too long? too short? follows structure?) is now quite accurate Top-down vs bottom-up evals Top-down: from the task description alone — what makes good output? Word length, action verbs, actionable takeaways. Claude is very good at generating these Bottom-up: discovered by reviewing many sample outputs — your gut vibes and feedback externalized into criteria. Claude is very bad at coming up with these. That’s all you. And it’s why they accumulate over time Peter’s podcast-takeaway skill is the live example: he has the top-down half (character-length checks, “understandable without watching the episode”), but the bottom-up half is where the question mark lives — is it exhaustive of all the feedback he’s given across every episode? His loop: run the skill → go back and forth → “reflect on our entire conversation and update the skill and evals so we don’t have to do this again” — with the honest worry that it overfits (one interview’s MECE complaint may not matter for the next) Three practical tips for eval-heavy skills Separate your evals into top-down and bottom-up inside the skill itself Fan out to sub-agents: with lots of criteria, give one sub-agent one criterion (or group) — give a model the whole list and it gets lazy and ignores things; focus it on one piece and it really focuses Have the AI write a spreadsheet / pivot table of criteria × pass-fail indicators, so you can see the hierarchy yourself and make judgment calls on what matters for this particular case The error discovery skill (the live demo) Open-source, free (link in the episode description) — invoked from Claude Code; it built the whole review interface from scratch in ~15 minutes, live on camera Five steps: Read the dataset and figure out its semantic type (article? code? traces?) Design a visual encoding — color, spacing, opacity (Gestalt principles) to show what varies in the data Build an interface — an HTML review app (Python backend); “so much better than me looking at my data in Google spreadsheets” Pick which samples you should look at — clustering, diverse initial sample Interactive loop — the agent watches your in-situ feedback via the monitor tool and proposes new samples or rubric criteria in real time Design philosophy: the human reads and gives open-ended feedback; the agent’s job is not to invent feedback but to group and distill it into actionable rubric criteria The writing demo: he reviewed AI-generated articles and gave taste feedback — “I don’t like negative contrast (‘it’s not X, it’s Y’)”, “I hate the list of threes”, staccato fragments — the agent annotated 361 suggestions across the dataset, and the most frequent failure mode was staccato fragments (he’d have guessed negative contrast) Live reflection beats reflect-at-the-end: interleaving human think time with AI think time, and the ~10-notes threshold works as a “carrot” that makes you actually read samples Once the rubric exists: turn it into a skill, one LLM judge per criterion, a dashboard, or live monitoring — the hardest part of evals is error analysis, and this automates the discovery half Bonus: how you eval something should inform how you design its interface — the same failure-mode annotations that power evals would make a great IDE that flags staccato as you write, instead of silently rewriting Do automated evals actually work? (Hamel’s benchmark) Vendor “auto-eval” tools (BrainTrust, Arize, LangSmith) promise: upload traces, chat with an AI, get your evals done Benchmark vs a human-annotated dataset: the tools recover a lot of the errors a human would — but all of them miss the same thing: errors that require product judgment and taste (e.g. a rental bot that doesn’t handle sales objections, or markdown leaking into text messages) Coding agents (Claude, Codex) performed about the same — the harness is thin; it’s someone else’s prompt The real benefit of the vendor tools is integration into your stack (traces in LangSmith → use LangSmith); precision is 80–90% best case, so 10–20% of “errors” found are red herrings — check recall AND precision, and sanity-check what the tool found Bottom line: automated tools get you a good baseline; manual review of the data is the edge — “actually reading stuff” is the edge, in evals and in code “There is no world in the future — even if you have AGI — if you’re building a product, you have to look at your data. You have to be able to inject your taste into the development of your product.” ...

Hamel Husain hosts Marek Galovic, CEO and co-founder of Top-K (ex-Pinecone data plane lead, ex-Shopify), on scaling multi-vector / late-interaction retrieval. ~24 minutes. Why single-vector embeddings fail agents Pooling is a lossy summary — it captures high-level semantics but drops the low-level detail precise queries need Agents issue many specific parallel queries; single-vector retrieval returns the same documents for all of them Agentic retrieval is sequential — noisy retrievals compound errors over multiple hops DeepMind’s limit paper: single vectors can’t capture arbitrary relevance matrices, even with infinite dimensions (embeddings are low-rank) Multi-vector = chop off the pooling layer Keep one embedding per token; score every query token against every document token (max-sim), then aggregate Preserves low-level detail; much better on out-of-domain and long-context retrieval Cost: 10-100x more storage, ~3 orders of magnitude more flops per score Existing workarounds (ColBERT-style compression into IVFPQ indexes) make updates and filtering hard in production Sparse multi-vector encoding (Top-K’s approach) Random projections map token embeddings into tens-of-thousands-dimensional space, then sparsify (keep top-k per token) Aggregating token-level sparse vectors into one document/query vector makes the dot product approximate max-sim Retrieval becomes inverted posting lists like BM25 — cost scales with non-zeros, not ambient dimension Two-stage: prune a billion docs to a few hundred candidates, then re-rank with exact max-sim (1-2 bit quantization, custom kernels, tens of thousands of docs/sec/core) Production numbers Sub-50ms P99 at billion scale; hundreds of QPS; 70MB/s writes with no query-latency impact Object storage as the durable layer + stateless compute; separate read/write pools Quality: a 100M-param multi-vector model outperformed an 8B dense model by ~40% on some video-doc retrieval; on BrowseComp an off-the-shelf 120B open model + multi-vector matched a proprietary GPT-5 setup; OfficeQA Pro went 18% @ $6/query → 42% @ $0.50/query Practical tuning advice Start with evals on your own private data, then hill-climb Dimensions are usually 128; you can prune tokens and quantize without hurting recall Relevance tuning (content score × user signals like distance/popularity) is underrated — e-commerce does it better than RAG teams “If you just vibe it, you can get better vibes from the system, but that’s not systematic. You need evals to know where you stand — and know if you’re improving or regressing.” ...

Hamel Husain hosts Will and Florian from Prime Intellect — the open-source reinforcement learning training team — on using evals to actually improve models. ~36 minutes. An evaluation has three parts Task set — your data, prompts, and scoring methods (what everyone focuses on first) Harness — the program that drives the LLM: Claude Code, Codex, or open-source harnesses like Prime / OpenCode Environment — where it runs: Docker, sandboxes, your own infra “If you are unable to express your task or your problem in any way or capacity, you’re also unable to improve your results.” ...

Shreya Shankar (Stanford, Bargain/LLM cascades) live whiteboards model cascades on Hamel Husain’s channel: cut LLM classification costs by ~90% with no fine-tuning and no accuracy sacrifice. 26 minutes. The problem: the Oracle plan Every record (e.g. customer support tickets) goes to one high-quality LLM (say GPT-5.6) → yes/no label (escalate to human or not) Great accuracy, brutal at scale — back-of-envelope: tens of thousands of dollars for a million tickets Goal: match the Oracle’s accuracy at ~95% agreement on your data, but much cheaper The fix: a model cascade Proxy model first: every ticket hits a cheap nano LLM, which returns a prediction AND a confidence score C(x) (0–1) Decision point: if C(x) ≥ threshold τ, accept the proxy’s label; otherwise route to the Oracle Confidence scores come free from every LLM API (token log probabilities); ground truth = “what the Oracle would say,” not absolute truth Step 1 — find the threshold (offline) Sample 200–500 tickets, run the proxy on each → (prediction, confidence) pairs Label each sample ticket with the Oracle Simulate the cascade for every possible τ = one of the observed confidences: record accuracy and cost Return the τ with minimum cost that still meets the target accuracy — sort ascending and take the first one that qualifies Trivial sanity check: τ = 1 routes everything to the Oracle → 100% by definition; the real question is how low you can go Lower τ = cheaper (more accepted by the proxy) — but the proxy’s mid-confidence predictions are where it gets wrong Step 2 — run the cascade Run the proxy on ALL tickets; accept every C(x) ≥ τ; query the Oracle only on the rest That’s the whole inference path Caveats and results Target accuracy on the full set is an estimate, not a guarantee — you measured it on a sample Statistical guarantees are possible via concentration inequalities; their open-source package/paper (“Bargain”) adds adaptive sampling — plain random sampling wastes effort on low-confidence points that are probably wrong anyway In practice: 90%+ cost cuts on a wide variety of classification tasks with just a nano model Cascades vs. model routers Routers are black boxes — no way to estimate the accuracy you’re actually getting Cascades are sample-driven and fit to YOUR data; the threshold search tells you whether any τ even meets the target Works for binary AND multi-way classification (well-calibrated confidence from logits over the class tokens); not for open-ended tasks — poorly calibrated probabilities there often mean no τ exists Don’t trust the nano model’s confidence a priori — that’s fine. The simulation either finds a threshold that meets the target or it doesn’t; “the procedure is not going to lie to you.” “You don’t have to have any confidence in the confidence.” — Hamel Husain, on why the simulation matters more than trusting the nano model ...

Zach Mueller (head of DevRel at Lambda, ex-Hugging Face) on how to think about open models — which ones, where they run, what they cost, and how to serve them. ~41 minutes on Hamel Husain’s channel. The big claim Open models are good enough for ~90% of queries from ~90% of people — the 10% exception is people pushing the frontier (OpenAI, Google, Anthropic, frontier research labs) Compared to a year ago, massive improvement in intelligence per parameter; small models now generalize, not just hyper-specialize The current open-model landscape Qwen non-MoE (27B, e.g. Qwen3.6 27B) — “the most capable model for an everyday agent”; 4-bit ≈ 14GB → fits a 16GB card; hundreds of people run Hermes agent purely on it Qwen MoE (~30B) — better than bigger Qwen dense models GLM-5.2 (~800B) — his daily driver for coding; needs 6×96GB cards (~$60k) or a big Mac DeepSeek V4 Flash — capable smaller model, a solid Haiku replacement Kimi (3T) — great for writing/review but 8×B200 ≈ 1.5TB VRAM MiniMax — strong but enterprise license changed since 2.6 Nemotron — US-built, US data; easier to get past enterprise “scary Chinese model” objections Speed targets (tokens/sec) 20 tok/s — baseline usable: background tasks, CPU offloading of a giant model 50 tok/s — usable workflow, roughly Opus-level; GLM 5.2 self-optimized its own deployment from 20→47 tok/s in a day (200–300M tokens through it in the first week and a half) 100 tok/s — ideal for sub-agents 750 tok/s (GPT-5.6 + Cerebras) — too fast to even review what the model writes; the joke is that becomes the chain of thought, presented at 50 tok/s Quantization FP8 native — most expensive; needed when RL rollouts must match training precision NVFP4 / MXFP4 — near-lossless; NVIDIA ships its own NVFP4 weights via quantize-aware training; Kimi K2 (1T) goes from 8–16 B200s in FP8 to fitting on 4 in NVFP4 His setup: GLM 5.2 NVFP4 on half a B200 cluster, Kimi K2.7 code on the other half Cost and security thinking Don’t point company data at OpenRouter — it routes to providers anywhere (including outside the country) and routers can carry “upload your code” instructions; even American providers (Grok) did it. If you can run it yourself, that’s the lowest-risk variable Buying hardware is usually a flawed equation — GPUs run 40–60% utilization, so “pays for itself at 100%” math doesn’t hold; try 2–3 month spot instances first, track token usage, then decide Self-hosting skills advance your career (his path: Accelerate at Hugging Face → GPUs at home) Serving vLLM — out-of-the-box experience, broad model support, an “oracle” that auto-picks kernels; home/single-node default SGLang — disaggregated inference, cache-aware routing for 100–1000 users; multi-node territory llama.cpp — fine for single-user local; he ignores it for batched/multi-user work Model routing and harnesses KB cache + shared context are everything — cold caches rebuild from scratch on every provider switch, so he won’t use “magic routers” unless he picks the models, the tasks, and stays on one provider Numena Ncode (XDR’s harness): a Claude Code fork with a model stack — Soul (driver) + GLM 5.2 (implementer/overseer) + DeepSeek V4 Flash (writes the code); XDR fine-tuned Kimi K2.6 for it and swaps models to measure how much code each frontier model removes Pi — minimal open-source harness; he replaced 3–4 Claude Code workflows with GLM 5.2 + Pi Amp — opinionated routing baked in (GLM 5.2 workhorse + a “Soul Oracle” for hard problems), API-priced; “the Puck” = an agent over agents Evaluating open LLMs: “vibes” plus trusted people (e.g. XDR); Lambda publishes the LLM index — deploy recipes (Docker) + tokens/sec benchmarks Codex side-note Codex’s computer use operates any app on your Mac; mobile support shows all running sessions/threads on your phone; and Codex can control Codex — fan out a project into 12–15 threads, let them talk to each other, orchestrate “Open weight models are good enough for about 90% of queries from 90% of people.” ...

Shreya Shankar (UC Berkeley PhD) presents DAB — the Data Agent Benchmark (“Can AI agents answer your data questions?”), from her PhD work at Berkeley. ~27 minutes on Hamel Husain’s channel. Why data agents matter Huge real use case: AI answering business questions — much office work is this Enterprises struggle to use Claude Code / Codex off-the-shelf for BI tasks, so they build their own data agents: Uber’s Query GPT (1.2M queries/month at launch), OpenAI’s in-house agent, Databricks, Snowflake Existing benchmarks fall short: Text-to-SQL — assumes clean data in one Postgres/Snowflake; real enterprises have fragmented data; SQL alone isn’t enough (need domain knowledge, reasoning over outputs) Table QA — context table + question; doesn’t scale to enterprise data volume The formative study — 4 real-world challenges Interviews with enterprises + Hasura (PromptQL) surfaced four properties missing from existing benchmarks: ...

Hamel Husain demos a Codex capability most people don’t know about: Codex can control itself — spawning parallel threads, letting them coordinate, and driving computer use. ~5 minutes on his own channel. The demo setup Working project: AEO — optimizing one of his websites for AI discovery (making it findable by AI search engines) Codex listed 16 high-value tasks from the project file (aeo-to-do.md) — the fuel for the orchestration demo Spawning threads Prompt: “Open a new thread for each task and explain how you’d tackle it, along with prerequisite steps” — Codex spawns 16 parallel threads in the sidebar Codex can also rename and delete threads itself Value: manage separate tasks completely independently, no window-jumping Threads talking to threads Inside any thread you can query another: “What is AEO 1 doing? Does it need any help?” Great for orchestrating when things get stuck, or starting a supervisor thread that manages others and unblocks them Steering and queues Ask for a status table when threads finish: which can run in parallel, which need human intervention or input Broadcast guidance to all threads: “Direct threads that can work independently with computer use to start — don’t start work if you need other threads to finish first” One thread inventories the active threads and coordinates the rest — “this starts to become super powerful” Computer use A thread opens the browser itself: checks Bing Webmaster Tools, Google Search Console, etc. It tells Hamel what it needs (accepting a verification), keeps going, and reports when it gets stuck Mobile Same thread list appears on your phone — manage all parallel threads remotely, even away from the computer “You can have Codex control Codex and become a power user to do a lot of things faster and parallelize your work.” ...

Joe Barrow (ex-Amazon/Adobe, ML lead at Pattern Data, now Adobe Research’s Document Intelligence Lab) on choosing an OCR model for AI document processing. 24 minutes on Hamel Husain’s channel. Why OCR matters Your app sits downstream of OCR quality — garbage in, garbage out, no matter what the LLM does after It’s not solved — Anthropic shipped bad PDF handling for a year because it was pulling text, not doing OCR; users noticed OCR is sticky — once you build on a vendor, swapping models is painful (Pattern learned this the hard way) Documents are evil — multi-column layouts, rotated scans, no reading order; TeX-compiled PDFs have no spaces (glyph glue), so naive text extraction gives you one run of characters The decision grid: two axes Text blocks vs. document structure Text blocks: word/line bounding boxes → grounding, evidence highlighting, cheapest Structure: headings, reading order, grouped paragraphs, tables, figure alt text, chart de-rendering → much better LLM input (LLMs are trained on markdown-like structure; raw line runs look like garbage to them) API vs. self-host API: ease of use, vendor support (startups retrain on your bad docs), minimal time — right for ~95% of teams Self-host: control throughput/concurrency (APIs cap concurrent docs — a real bottleneck), stable weights, domain fine-tuning, no lock-in, cheaper at bulk — but only if your time ≈ $0 or you run huge batches The four quadrants Big cloud APIs (AWS Textract, Google Cloud Vision, Azure) — $0.60–1.50 / 1k pages; word+line boxes only; tables/forms a la carte at $10–15 / 1k Document startups (Reducto, Data Lab, Extend, LlamaIndex) — $5–20 / 1k pages, “fast” vs “accurate” tiers; structure included (markdown/HTML, tables, figure boxes) Open pipelines (PaddleOCR, Nemo Tron, Tesseract) — 10–100M params, nearly free, edge-deployable (PaddleOCR runs on phones/e-ink); text lines only, post-process with layout models Open VLMs (LightOn OCR 2, GLM OCR, GOT-OCR, Chandra/Surya) — 600M–8B params, native document structure, ~$0.20–0.30 / 1k pages on a saturated H100; hallucination risk exists but clouds hallucinate on crusty scans too How to actually choose Ignore benchmarks (OmniDocBench, CR Bench) — they’re not run on your data Build a 50–100 page sample of your own representative PDFs Run a few candidates, diff the returned text (catches junk-on-handwriting fast), visualize the boxes ~a day of effort total — then pick Watch the license Chandra/Surya (Data Lab): free only if org < $2M revenue AND not competing with Data Lab LightOn OCR: Apache. GLM OCR: MIT (but relies on PaddlePaddle’s Doc Layout model — Apache — both apply) Self-hosting, for the ~5% who should Inference engines: VL (default, OpenAI-style client) or SGLang; infra: Modal (request-queue scaling beats SageMaker), BaseTen, or big cloud for one-off batches 1B-param models (LightOn, GLM) on H100 → ~10k pages/hr, 20–30¢ / 1k pages; 4×3090 ≈ one H100 → 3–4 pages/sec His 7M-page local-laws dataset: ran over a weekend at ~30¢ / 1k all-in “You can process 1,000 pages per second, but it doesn’t matter if they’re all wrong — then your entire app’s output is going to be garbage.” ...

Lucas Rocha, an engineer at Brazilian edtech Nova Escola and alum of Hamel Husain’s AI Evals course, tells his evals rollout story backwards on Hamel’s channel — 34 minutes on going from a messy spreadsheet to four calibrated judges running in CI and on daily production samples. ...

Abi Aryan (“Inference Infra Queen”, Maven inference-engineering instructor, NOVA IMS) teaches LLM latency from first principles on Hamel Husain’s channel — a 41-minute Colab demo that separates prefill from decode and shows why decode owns your latency and cost. Where the time actually goes Opening puzzle: the same TinyLlama model, same T4 GPU, same inference code produced one request at 3.3s and another at 8.8s — no tricks, no patching Most people’s mental model (“more tokens = more compute”) is wrong; the request shape decides where time goes The inference shape = 4 variables: input tokens, output tokens, batch size (concurrent requests), context length (what attention can see) Real workloads cluster into 4 shapes: chat (~1:1), RAG (~20:1 input:output), creative writing (tiny prompt, long output), agentic (no fixed ratio — the hardest to predict) Agentic shapes are global, not static: the request at step 6 contains every token from steps 1–5 Prefill vs decode: two different bottlenecks Prefill: whole prompt in one parallel forward pass, produces the KV cache; compute-bound and GPU-saturating; scales with input tokens; experienced as time-to-first-token Decode: autoregressive, one token at a time; memory-bandwidth-bound — every token streams the full weights + growing KV cache (2.2GB of HBM traffic for one TinyLlama token); scales with output tokens; experienced as streaming speed “A GPU is just not another faster CPU” — the two phases have different bottlenecks, so different optimization strategies TTFT and streaming speed are reported as separate numbers because they come from different phases The Colab demo: plain Transformers, no engine Deliberately runs Hugging Face Transformers in eager mode — no vLLM/SGLang — because vLLM hides the prefill/decode split (and doesn’t install cleanly on Colab); seeing the raw loop first is what makes the engines legible TinyLlama 1.1B in FP16 = 2.2GB VRAM on a T4; the inference function syncs to the GPU clock (torch.cuda.synchronize) and times prefill vs decode per request Smoke test first (22 in / 8 out tokens): prefill 152ms ≈ 144 tok/s, decode 7 tok/s — same model, same GPU, different phase, different bottleneck Always warm up the GPU before experiments — skipping warmup in production is a classic footgun The three shapes, measured Short-in/short-out (chat): prefill ~40ms, decode dominates — prefill ≈ 3% of latency Long-in/short-out (RAG): prefill share grows to ~16% — the only shape where prefill visibly matters Short-in/long-out (creative writing): prefill ≈ 0.5% — and this shape has the longest total latency, because decode is sequential Read vs write asymmetry: ~1,205 input tokens prefill at ~0.44ms/token — writing a token costs roughly 300x more GPU-clock time than reading one That’s why pricing pages price input and output tokens separately Small requests waste the GPU Prefill throughput: 143 tok/s (25-token prompt) vs 2,271 tok/s (long prompt) — a 16x difference in the same phase A tiny request leaves the GPU nearly idle — the observation that birthed continuous batching (vLLM/SGLang’s scheduler.py batches as many prefill tokens as possible into one request) KV cache: growth depends on architecture KV cache = memory growing with output tokens; ~0.17MB per token for TinyLlama, measured under theory Growth is architecture-dependent: TinyLlama uses GQA (KV pairs shared across a group of heads); MQA/MLA compress differently; MoE models (e.g. DeepSeek) generate less KV cache When picking a model, prefer architectures that group the KV cache — “think about this as the context filling” Agentic loops: where cost really lands Simulated 5-step agent loop (build prompt → inference → append to history), no framework — frameworks add latency, hide token consumption, and fail fast in production Every step re-sends the full history from scratch (no context graph, no memory card) — runtime grows ~O(n²) in input tokens Result: prefill ≈ 3% of agentic latency; decode accumulates every step; cumulative latency hugs the decode line Agent shapes add three more variables: batch size, context window, number of tool calls (more tool calls → more latency) Costed on OpenAI pricing: a 5-step loop ran up 11.9x the cost of one chat request — almost all from re-sent input tokens feeding decode The agent-optimization target: the smallest amount of KV cache you can share back to do the decode The dense takeaways Latency lives in decode; output tokens are ~2 orders of magnitude more expensive than input tokens TTFT is a prefill story; total latency is an output-length story — cap max tokens, don’t just tune the retrieval pipeline Small requests waste the GPU — batch (143 → 2,271 tok/s) or let the engine’s scheduler do it Agents rebuy their entire history every step — context management is survival, not optimization Homework Try batch sizes 4 and 8 vs 1 and watch prefill/decode latency change Swap engines: llama.cpp on Mac (auto-batching/scheduling), vLLM/SGLang on Linux — how does each affect the phases? Try INT8/INT4 quantized versions of the same FP16 model — memory, latency, cost per phase Explore prefix caching: smallest KV cache you can hand back; read vLLM’s batch.py and the paged/flash attention kernels Q&A: production mistakes and scaling Mistake #1: shipping models with no smoke test/warmup before pods report ready Mistake #2: serving the same model on different ports without KV-cache sharing across them Open problem: how long to hold KV cache before eviction — dead weight, the same question database companies faced Self-hosting economics: hybrid approach — small one-off requests on API GPUs, long-running multi-step agentic systems on your own GPUs (EU regulation is the deciding factor in Europe) Speculative decoding & chunked prefill: built to control KV-cache growth on very large models (~45B+); not a major concern for small ones Resources: her Maven courses (forward-deployed; inference engineering & systems design), gpengineering.com with her PMPP notes, plus intro and kernel-engineering books “Context management… that’s what will make the difference between whether you’re building a system that would survive or whether you’re building a system that will eventually get killed.” — Abi Aryan, on agentic systems ...

Hamel Husain makes the case that “are you building the right thing?” matters more than evals: if users can’t verify an AI answer, your product creates work instead of saving it. Three worked examples and four design principles. ~17 minutes on his own channel. The core problem: verification is the bottleneck Classic AI product: user asks “what was revenue for product A last quarter?”, AI queries a database and returns “$4.21M” — but the user can’t validate the number without redoing all the work Lenny’s tweet: data science teams now spend their time reviewing half-assed AI analysis from PMs and data engineers — 50% of the time it’s wrong Creating output is now easy; verification is the bottleneck — design products with that in mind If a product seems impossible to eval, that’s a product smell: your users have to check it too Example 1: business Q&A → evidence-backed analysis Instead of a bare answer, show supporting information: base the analysis on already-vetted analysis (notebooks others signed off on) State the assumptions — confirm the metric definition (from a semantic layer, linkable to the governed definition) Show intermediate calculations (returns, customer counts) and raise issues the user might care about “Open as notebook” UI idea: an AI-generated notebook (Jupyter/Marimo style) with narrative + queries — step through it, edit it, ask AI questions about it, and have the AI state what it couldn’t verify Put yourself in the expert’s shoes: a data scientist checks prior analyses, definitions, and intermediate calcs — give users those same affordances Real product doing this today: Hex — a chat interface that flows into a notebook; progressive disclosure is an established pattern we forgot in AI apps Example 2: PE lesson plan assistant (K-12) Vanilla version: input grade, class length, location, equipment → AI spits out a lesson plan — no way to judge quality Better: anchor the output in existing curriculum — what are trusted colleagues doing? “Here’s a lesson plan exactly like yours, used at 14 schools, run 30 times, by teachers you trust” Show the edits made for your inputs (“shortened to a 45-minute class, matched to your equipment”) — accept or reject each Cold-start problem: seed with expert-vetted plans up front; it makes the app easier to eval and users more confident Example 3: medical claims report generator The task: read a patient’s chart + claim (thousands of documents) and produce a 52-page report supporting or denying a claim — a doctor can’t eyeball 52 pages Redesign as a research assistant: surface atomic elements first — contradictions (open both pages, resolve), key facts (validate, include, dismiss), open questions (add notes) Do the work the user would normally do and guide them through it; understanding built along the way is arguably more valuable than the final report Four design principles Provenance — show where information comes from (prior analyses, source documents, other lesson plans); the more curated/trustworthy, the better Progressive disclosure — show details at the right time; adapt as your product and users mature Signals and heuristics — mirror how experts sanity-check: other reports, smell tests, social proof, supporting evidence Modularity / small steps — break work into small verifiable pieces, show your work along the way, and sometimes force the human in the loop (e.g. doctors step through findings before the final report) “The goal is to make the human understand, not necessarily to generate a report.” ...

Hamel Husain demos the Codex desktop app’s remote-control mode — the best setup he’s found for untethering AI coding from your computer and running sessions from your phone. ~4 minutes on his own channel. Why remote control When coding with AI, don’t be tethered to your laptop — walk around, close the screen, check things on your phone The Codex desktop app is the best client he’s found for remote AI coding Setup: connect your computers Settings → Connections — he’s connected to a Mac mini and a MacBook Pro (both servers for agentic coding) To add: go to the server, click the link to get a code, then Add it in the app Keep the connection dots green — refresh occasionally; rarely you may need to delete and re-add a connection Keep the Codex app updated (Codex menu → check for updates) Starting a remote session New chat (Cmd+N) or Add new project → Remote, then pick the connected computer and give the folder path You see all sessions running remotely — which machine, which folder, live green dot Live demo Prompt: “Look at my Gmail and find the last 10 emails received from this email address, which are newsletter posts. Put it in the table” — uses the Gmail skill/connector and computer use on the MacBook Pro From the phone (ChatGPT app → Codex tab): see every connected computer and live session; send follow-ups like “Please critique my writing in all 10 emails with subagents. Use the writing skills in this repo to do so” Desktop shows the run: 5.5 High extra fast model, summary view listing the 4 spawned subagents processing email batches; phone shows 2 agents running / 2 finished You can even attach photos or screenshots from your phone “It’s the best remote control I’ve ever used. I haven’t found any limitations.” ...

Shreya Shankar (Stanford CS professor, co-creator of the AI evals course with Hamel) kicks off the 12-part AI product engineering series. 27 minutes on Hamel Husain’s channel. Why this matters Output quality is the biggest barrier to productionizing agents (LangSmith annual report) — and figuring out how to evaluate models is genuinely hard Vendors (LangChain, Braintrust, Arize) are selling end-to-end automated eval tools: point an LLM at your traces, it finds and fixes your bugs The catch is epistemic: what “good” means lives in your head, not in the traces — if a tool could fully fix your product, it could fix everyone’s, and there’d be nothing left to differentiate yours AI’s real job: help you express and apply your judgment faster, not replace it The eval lifecycle (analyze → measure → improve) Error analysis — the hardest step: take traces and find failure modes. No perfect definition of “mistake” (you can’t define slop, but you know it when you see it) Measure — how prevalent is each failure mode? Pareto applies: ~80% of issues come from ~20% of failure modes — prioritize those Improve — fix the product: prompt instructions, model switch, fine-tuning. Iterate forever AI is weak at the front (taste-specific error analysis) and strong at the back (measurement, prompt optimization, hill-climbing). ...

Hamel Husain, a decade-plus AI consultant himself, argues the hottest job in the economy is usually a bad hire — with two exceptions. ~3 minutes on his own channel. Why hiring an AI consultant usually fails The problems you had before — talent, organizational, skill issues — are still there when the consultant leaves AI’s whole point is to remove the middleman: you express what you want in natural language and inject your domain knowledge directly into the tool If they just implement and walk away, you can’t maintain what they built — worst case you keep paying them, i.e. prompting a consultant instead of prompting an AI His track record: in over a decade, third-party AI implementations he’s seen don’t work When a consultant is worth it Education: the consultant up-levels your entire team and makes themselves obsolete — you keep the skills after they’re gone Skin in the game: they don’t get paid unless they affect your bottom line (increase revenue or decrease costs), demonstrated at a quality bar — aligned incentives The alternative Learn AI yourself — it’s the most accessible technology ever created; you mostly need to learn how to talk to it Coding skills help but aren’t required for lots of things; a growth mindset makes effective use tractable with practice “Instead of prompting an AI, you don’t want to be prompting a consultant.” ...

Hamel Husain answers a client question — how to eval an AI PE curriculum planner for K-12 — with a one-minute twist: evals and product trust are the same problem. ~1 minute on his own channel. The eval problem A client building an AI curriculum planner for K-12 physical education couldn’t figure out how to eval it The answer: think creatively — put yourself in the educator’s shoes Questions an educator actually asks Was this plan created by an expert? What other teachers are using this plan? Is this plan based on best practices and expert knowledge? The takeaway Infuse those signals into the product — they give the end user trust in what they’re seeing That same design doubles as your eval: provide users a way to assess what they see, and you can assess it too If you can’t eval your product, your product probably sucks “If you can’t eval your product, your product probably sucks.” ...

Hamel Husain on his own channel — a 1-minute clip on the “evals take too long” objection and why your users eval your product whether you do or not. The “evals take too long” objection One of the main reasons people skip evals: reading a trace takes forever — a lot of human hours you “don’t have time to do” His example: an AI data agent that answers business questions (supply, revenue) — “the most popular internal tool I’ve seen so far” The question: how do you eval that? If you can’t eval it, it probably sucks One principle to keep in mind: if you can’t eval your product, your product probably sucks — because your users need to eval the product When you present an AI answer to a user — say, an answer to a business question — you need to figure out a way to convey trust “If you can’t eval your product, your product probably sucks. Why is that? It’s because your users need to eval the product.” — Hamel Husain ...

Hamel Husain on his own channel — a 1.5-minute clip on LLM-as-a-judge evals: how to measure the judge’s noise and why a judge you can trust 100% of the time doesn’t exist. Measure the judge against ground truth An LLM judge is a “soft check” — you can scientifically measure how good it is by comparing it to labels Assemble a dataset of human labels as ground truth (is this thing good or bad?) and measure how much noise the judge has Even ~100 labels is enough to measure the noise There are different kinds of noise — you can tune the judge to reduce it, but there will always be some A judge is a black-box classifier There’s no such thing as a judge you can blindly trust with no noise at all It’s a classifier, like a stop-sign detector: get it very reliable and there’s still an edge case somewhere producing false positives and false negatives You have to be okay with that Tolerable noise is a business decision Tune the judge to reduce noise, then make the call: is this level of noise tolerable for your use case? Bake the eval’s known characteristics into your decision making — that’s the best you can do with an LLM judge “Even if you can get that to be very reliable, there’ll always be an edge case somewhere where it’s going to give you a false positive, false negative.” — Hamel Husain ...

Hamel Husain on his own channel — a 2-minute clip on why traditional scoring metrics (ROUGE/BLEU) and rating scales make evals harder to act on, and why he defaults to binary pass/fail. ROUGE/BLEU: string similarity from a different era These scores come from traditional ML/NLP — at their core they measure string similarity A coarse metric that made sense when LLMs could barely produce coherent language; “we’re way past the point of producing coherent English” now Part of the “eval industrial complex”: we have metrics, trust us because we’re experts — it feels fancy but isn’t what you need Rating scales hide uncertainty Unless you have a very sophisticated setup and put real resources into aligning scores with humans — which ~99% of teams don’t — a 1-to-10 or 1-to-5 scale is a bad idea Nobody knows what 4.2 versus 3.7 means Humans just hide their uncertainty in the middle values, so you don’t get good decision making out of it Binary pass/fail is what you can ship on At the end of the day you have to ship your product: is this good or not? Binary evals are easier to align with humans — which you always have to do with an eval — and easier to action A score of 3.2 → “I don’t know what’s wrong.” A fail → “okay, it failed, now you can action on that” “You see a score of 3.2, you’re like, I don’t know what’s wrong… but fail, it’s like, okay, it failed, now you can action on that.” — Hamel Husain ...

Hamel Husain on his own channel — a 40-second clip on why rating-scale evals (1–5) are the wrong default: he pushes for binary pass/fail evals with failures scoped into specific, actionable criteria. Binary beats a 1–5 scale He tries “really hard” to make every eval binary — pass or fail Scope the failure: “agent took too long → failed because it took too long”, “too many steps → failed because it took too many steps” He has never gotten stuck converting a 1–5 scale into binary, across all the companies he’s worked with The exception: a well-tested rubric everyone is confident in — then a 1–5 scale might genuinely work In most cases the scale just “kicks the can down the road” — the noise doesn’t disappear, it gets hidden “In a lot of cases, it just kind of kicks the can down the road, and you’re just hiding a lot of noise in there.” — Hamel Husain ...

Hamel Husain (founder of Parlance Labs, co-author of an upcoming book on AI evals with Shreya Shankar) on the Applied Intelligence podcast — 39 minutes on why evals are the testing framework most businesses skip, and how to think about them the right way. What an eval actually is An eval is structured data analysis and debugging on your AI application — you know what’s broken, and you know what to prioritize fixing It’s the answer to a real problem: AI outputs are stochastic (text), with no deterministic “right or wrong” to test against Software testing assumes a finite surface of failure; AI has an infinite one, so you have to decide what to measure Generic metrics are a trap Search “how to evaluate my AI app” and you’ll find vendors promising a dashboard of helpfulness, conciseness, toxicity, coherence scores (usually 1–5) Nobody knows what those numbers mean, they don’t correlate with what matters, and they actively burn engineering cycles looking at metrics that don’t matter “Reduce hallucination, increase helpfulness” is the same generic-metrics mindset — you’re going through motions without knowing what’s actually wrong There’s no easy button; you can use coding agents to write the evals, but you still have to be thoughtful about what you measure Start bottom-up, not top-down The process starts with error analysis: a blend of qualitative and quantitative work to find what’s broken in your application Most people are too top-down (worried about failure types they imagine) and get lost in generic metrics; the bottoms-up read of your actual data is the missing half Evals are really a process of eliciting your specification, then measuring against it — you can’t know “good” until you look at outputs and iterate AI can’t do your evals for you AI is great at fixing deterministic bugs; it cannot read your mind — it doesn’t know what “good” feels like for your product The future is AI walking you through the evals process and interrogating you, not doing it alone — a human stays in the loop Eval tooling needs to render your data domain-specifically (images as images, emails as emails, chat as chat), so it’s more than a chatbot Two failure modes Hamel sees most Not using AI deeply yourself — no coding with AI, no building with it → bad intuition, bad specifications Reaching for complexity too fast — day-one orchestration frameworks, graph databases, multi-agent setups before you can reason about what’s happening Deliberate, not slow AI lets you build the wrong thing faster — and many wrong things faster; it also lets you build the right thing faster You need product sense, grounding, and taste, or you just churn through bad ideas at higher speed AI amplifies who you are: if you’re okay with slop, it removes the friction and amplifies the slop Chatbot vs MCP Slapping a chatbot on an existing product is the mediocre default when there’s a “we need AI” mandate from the top Better: expose an MCP or an API on your product — the Google Workspace CLI is the canonical example of how much more useful agent-friendly access is Interface matters: a scheduling flow shouldn’t be a brittle text back-and-forth when a picker widget gives visual confirmation and avoids bugs Guardrails A guardrail is a specific eval sitting in the request/response path that blocks a bad output (competitor talk, profanity) Off-the-shelf guardrails are just someone else’s prompt, tuned to a different domain (shopping, travel) — you still have to do the evals work to know which failures to guard against Prioritize failures you can simulate or actually observe; unobservable ones are lower priority The stack, the vendor, and the team Start with the most powerful model you already know, build an eval harness around the metrics that matter, then back off to smaller/cheaper models and reason about the latency–cost–quality tradeoffs Data science gets more important, not less: the ability to ask the right questions is directly proportional to the quality of output you get AI competency is core to any knowledge-work business, so be careful outsourcing it — use third parties to upskill your team (a deliberate training exercise), never as a crutch The takeaway Parlance’s model is a “driving school” boot camp: they pair-program the whole end-to-end evals process on your data until you don’t need them The million-lines-of-code demos are all backed by harness engineering — metrics, logs, traces, observability — and evals are almost all of that harness You don’t need an R&D budget, maybe just a token budget: a $100 plan and deliberate experimentation gets you a long way “AI cannot read your mind. It doesn’t know what you feel like good is.” ...