Hamel Husain and Angelina Yang design an evaluation workflow for a hypothetical podcast assistant in this 44-minute TwoSetAI interview.

Start with a product problem

  • A sole user who reads every output already provides an informal evaluation loop; systematic tests become more valuable when other people use the product.
  • Read real interactions and describe specific failures before adopting generic helpfulness or coherence scores.
  • An obvious missing document or broken tool may call for a direct fix; uncertain improvements that need experiments are stronger candidates for evals.

Make the evidence easy to inspect

  • A trace records the task, prompts, tool calls, retrieved material, and results, including work hidden from the user.
  • Combine random exploration with customer complaints, thematic sampling, unusual sessions, and missing requested outcomes.
  • Build a review screen that renders artifacts naturally: emails should look like emails, with recipients and CC fields visible.
  • Start with 100 traces as a practical goal, then continue until new examples stop revealing new patterns; the number is not a statistical guarantee.
  • Read from the start and focus on the first major failure, since early mistakes can drive later symptoms.

Turn observations into useful checks

  • Write clear notes about the user experience before getting absorbed in root-cause debugging.
  • Count failure categories, then prioritize using both frequency and product importance.
  • Prefer real user data; synthetic examples need explicit variation and a hypothesis about what might break.
  • Choose among ordinary tests, classifiers, and LLM judges according to the decision and maintenance cost.

Evaluate the evaluator

  • Use labeled examples in separate training, development, and held-out test sets.
  • Training examples can become prompt demonstrations; the development set guides improvement, and the test set checks generalization.
  • Prefer pass/fail criteria when they match the product decision, and measure how the judge agrees with human labels.
  • Read generated judge prompts for oddly specific exceptions that merely memorize individual examples.
  • A calibrated judge can flag production failures and guide coding-agent improvements, but the optimization loop still needs oversight.

Keep the suite useful

  • Add new challenges as the product changes; review false alarms and failures the existing checks miss.
  • Saturated, expensive checks may run less often, while important regression protection remains.
  • Report category-level pass-all rates with an explanation of the failures underneath them.
  • Track durable product measures separately: changing evaluation sets can make month-to-month score comparisons misleading.

Keep tools and responsibility proportional

  • A small project can start with stored traces and a custom annotation screen; larger systems may need stronger search and access controls.
  • Human judgment still defines good behavior, reviews categories, chooses priorities, and checks important prompts.
  • Husain describes software subscriptions as buying accountability: someone else maintains the service and owns its failures.

“Always read the prompts.” — Hamel Husain

Caveat: This is a design exercise, not a deployed podcast-product study. Its example user counts and automation impressions are illustrative.