Hamel Husain interviews Teresa Torres — author of Continuous Discovery Habits and the coach who popularized the Opportunity Solution Tree — ~91 minutes on why most teams build the wrong thing, what “talk to your customers” actually requires, and how evals are the missing discovery habit. Hamel’s audience is engineers, and Teresa’s audience is product teams; the conversation is the overlap, ending in one of the best eval war stories Hamel says he’s heard.

“Talk to your customers” is usually done wrong

  • Discovery vs delivery (Marty Cagan’s framing): deciding what to build vs actually building it. Most companies treat all the work as delivery and never measure impact — so they never learn their ideas failed
  • Discovery, at heart, is fast feedback loops — the scientific method applied to “is this the right thing to build?”
  • Red flag for bad discovery: you’re not constantly throwing away or evolving your ideas
  • Confirmation bias is unconscious: your brain filters out disconfirming evidence before you notice it — so you must design feedback loops to be disconfirming from the start, not just “go get some feedback”

Ask for stories, not opinions

  • The founders’ classic — “hey, what do you think of my idea?” — is a terrible experiment: most people aren’t your target customer, and non-customers’ opinions are garbage
  • Ask “tell me about a time when…” — a specific instance, walked through from start to finish. General questions trigger System 1 (fast, lazy) answers that don’t match actual behavior
  • The jeans story: a woman says she buys on fit, price, and favorite brand; asked about her last purchase, she bought her usual brand on Amazon because it was on sale — she’ll compromise fit for a deal. People say ≠ people do, and often people think they do ≠ what they do
  • Collecting a rich story is a practiced skill: situate the person in the moment, build the timeline (“what happened next?”), redirect generalizations back to the specific instance

Assumption testing: test the pieces, not the idea

  • Break the solution into steps (story mapping), then surface the underlying assumption at each step
  • Assumption types: desirability (do they want to?), willingness (want ≠ willing — the sketchy-ATM example), usability, feasibility, viability, ethics, sustainability
  • A single assumption can usually be tested in a day or two — the whole idea can’t

The Opportunity Solution Tree

  • Born from a real fight at an alumni-community startup: an engineer (Seth) stayed up all night building a Google Maps “where alumni live” map while Teresa, the PM, wanted rebalancing give/receive — they were arguing in the solution space without agreeing on which opportunity mattered most
  • Shape: outcome at the root → opportunities (unmet needs, pain points, desires — gathered from interviews) → solutions. Keeps everything coupled to a real need and to the business outcome
  • Inspired by Anders Ericsson’s Peak: experts have mental representations novices lack; the tree is Teresa’s representation made explicit
  • Why engineers matter here: they know what’s just now possible (her podcast is literally called Just Now Possible) — and with AI, engineers are finally engaging with discovery as companies push product-mindedness on them

Evals as a missing discovery habit

  • Teresa’s AI products grew out of deliberate practice: an AI interview coach that grades story-based interviews on four elements (story-based question, set the scene, build the timeline, redirect generalizations) and pulls excerpts showing what you did well/wrong
  • An AI participant removes the embarrassment barrier — practice in private, then get graded
  • Goal: infinite practice with expert feedback — and measurable learning outcomes (do scores go up across rounds?)
  • How she knew the coach was good: cohorts capped at 50, so she QA’d every response by hand → realized that didn’t scale → took Hamel’s evals course → error analysis → evals to systematically reduce the coach’s mistakes
  • “Evals are just another feedback loop” — why she calls eval a missing discovery habit

AI synthesis: no magic pills, show the work

  • Dumping transcripts into Claude or NotebookLM gives shallow, ungrounded results unless you’ve designed the research with explicit goals
  • Framing is everything: “I get too many Slack notifications” (what customers say, what Claude captures) closes the solution space to notification fixes; the real need — “don’t interrupt me during focused work” — opens it. Claude never makes that leap
  • “Holy crap, Claude is already better than the average product team” at the cognitively hard synthesis work — disturbing, then motivating: Teresa realized she could teach the AI to be better than most teams, and that working with her AI raises the ceiling for experts while raising the floor for novices (research on AI + expertise)
  • Teaching shifted from transmitting knowledge to creating spaces to learn: AI maker studios, AI show-and-tell, community practice — “you only learn in community”

The Vistily build: four AI services + change sets

  • Teresa licenses her AI services to Vistily (she’s the AI researcher; they run the company, compliance, sales). Deployment is a source-access partnership — they run her software in their AWS accounts for data-residency/compliance reasons
  • Service 1: classify the transcript (is it usable?). Beta reality check: 500 teams uploaded 3 interviews each — only 8 were story-based. They relaxed to general interviews with a strength-of-evidence model: story-based signals are strong, mini-stories inside general interviews can be too
  • Services 2–4: extract key moments + opportunities → generate a tree from scratch after 3 interviews → update the tree via change sets, preserving human edits
  • Change sets are the LeetCode-hard part: splits/merges with source provenance are ambiguous. A deterministic git-style diff failed; the fix was having the LLM emit a scratchpad of its moves as the change set — which also improved agent quality (chain-of-thought-like) and enabled an audit-tool agentic loop (apply the change set, verify output matches, kick errors back)
  • Economics: a fully agentic version ran ~$25/run; the pipeline with agentic steps is 50–60¢/run. Teresa is experimenting with model cascades after Hamel’s lightning lesson

Three weeks on one error — an eval war story

  • A customer loved her first AI-generated tree — “you found everything the agency found and more” — then flagged one branch: “a lot of unsorted opportunities… I’m going to have to come back and clean this up.” That’s the failure mode: missed groupings
  • The LLM judge wouldn’t calibrate. Root cause: confounding upstream errors — poorly framed opportunities that no customer complained about but that confused the judge. Teresa resisted fixing them (“I’m wasting my time on something good enough”); Hamel’s advice: attack the upstream errors
  • Fixing the upstream error exacerbated the target error and vice versa — two evals counterbalancing each other, four big experiment variants over three weeks
  • Resolution was not a prompt fix: she ran her evals as guardrails in an agentic loop — the agent gets kicked back and corrects on the second loop
  • Her detailed account is her next blog post (producttalk.org — she also flags the free “Product Discovery Basics” article and Continuous Discovery Habits)

“Holy crap, Claude is already better than the average product team.” — Teresa Torres

“You need to focus on the upstream error — and everyone resists that.” — Teresa Torres

“Teaching was never about transmitting knowledge. My job is to create spaces for people to learn.” — Teresa Torres