Teresa Torres makes a useful case for treating evaluations as a product-team discovery habit, not an engineering afterthought. An eval is simply a way to measure whether an AI workflow is doing what we need—but because LLMs are probabilistic and often produce semantic output, the team first has to define what “good” means in context.

Start with the errors

The practical starting point is error analysis: inspect a range of outputs, record what went wrong, group mistakes into recurring failure modes, and prioritize the errors that matter to customers. Torres’s examples are deliberately concrete: a hallucinated quote can be caught with a deterministic check, while a leading interview question requires judgment.

For a personal workflow, one deeply reviewed example may be enough to identify the right checks. A customer-facing product needs ongoing review of production traces—the detailed records of inputs, prompts, tool calls, intermediate steps, and responses.

Choose the simplest measurement

Torres describes four common eval types:

  • Golden dataset: known inputs paired with ideal outputs; useful for small tasks with one correct answer, but tedious and limited for large or open-ended work.
  • Code assertion: a cheap, deterministic rule such as checking whether a quote appears verbatim in a transcript.
  • LLM-as-a-judge: a second model evaluates a semantic quality; keep the criterion simple and binary, and expect judge error.
  • Customer feedback: explicit ratings or implicit signals such as regenerating, editing, or asking a follow-up question.

Her default is to try code first, use golden datasets to isolate smaller subproblems, and reserve LLM judges for cases that genuinely require judgment. Combining them can reduce cost: a cheap structural check can filter the cases that need a more expensive semantic judge.

Build a baseline, then experiment

Define representative inputs, run the current workflow, and score its baseline error rates. Then change one thing—a prompt, model, or orchestration pattern—run the same inputs and evals again, and compare every category. The loop is deliberately familiar: measure, experiment, compare, repeat.

The important discipline is not outsourcing correctness to an eval vendor. Correctness depends on the product’s context, so the product team has to supply the judgment. Evals make that judgment repeatable and expose whether changes actually improved the experience.