Hamel Husain walks engineers and product managers through AI evaluation take-homes using a hypothetical tree-care app (21:18, including Q&A).

Start with the data, not the coding agent

  • Spend 50–70% of the time understanding the data, Hamel recommends; inspect actual examples rather than asking an agent to discover every error.
  • His fictional Tree Buddy app combines input checks, species identification, data extraction, and care recommendations—each needs a different evaluation.
  • Find missing labels, inconsistent units, ambiguous categories, and examples that do not match the product’s intended use.
  • A guardrail dataset containing only allowed inputs can yield 100% accuracy while testing nothing about rejecting bad inputs.
  • Use trustworthy external data where appropriate; synthetic negative examples can fill gaps, but remain proxies rather than proof of real-world coverage.

Run experiments you can explain

  • Begin with a simple prompt-only baseline, then compare targeted changes such as retrieval, a different model, or prompt optimization.
  • For each feature, record the goal, method, results, and what the comparison taught you.
  • Keep development examples separate from held-out test examples so repeated tuning does not masquerade as general improvement.
  • Compare quality with speed, cost, and complexity; the highest score need not produce the best product.
  • The model names, dataset counts, and result rows in this example are illustrative—not benchmark findings.

Challenge weak metrics constructively

  • If asked to combine one-to-five helpfulness and tone ratings, provide the requested calculation and explain its limitations.
  • Propose concrete checks: is the photo sufficient, is watering advice grounded in evidence, and does the response avoid blaming the user?
  • If an AI grader cannot be checked against human labels, explicitly say its scores remain unvalidated.
  • Explain problems politely: a weak assignment may be a deliberate test of judgment, or simply a poorly designed interview.

Make the submission easy to review

  • Structure it as executive summary → data → experiments → results → learnings and limitations → next steps → AI-tool disclosure.
  • Keep the executive summary to roughly a third of a page; synthesize the winning approach rather than repeating every experiment.
  • Name the exact additional data and product changes you would pursue—not just “get better data.”
  • Follow the requested PDF or repository format; a flashy generated UI is not a substitute for a readable report.
  • PMs should demonstrate product judgment and data literacy; coding agents and eval skills are useful only when the candidate keeps thinking.

“Never assume a data set you are given in real life and especially in an interview is quote correct.”