Hamel Husain’s 15-minute walkthrough compares decision models, which return classifications and scores, as judges for AI evals: tests of an assistant’s behavior.

Start with labeled failures

  • The experiment uses apartment-leasing assistant traces covering bookings, resident requests and handoffs to people.
  • It compares Jev 1.13, Perplexity Decider 1.1, Liquid d1 and d1-ultra, Cloudflare Clef Flash and OpenAI Decisions.
  • The companion article clarifies that performance uses 98 traces after two internal QA examples are excluded.

Pick an operating point deliberately

  • Precision measures how many flagged traces really failed; recall measures how many known failures the judge catches.
  • Set minimum requirements before optimizing a score; the illustration asks for at least 80% precision and 80% recall.
  • Tune thresholds on training examples and measure on held-out examples, using the same stratified five-fold splits across models.
  • Choose F1 for balanced precision and recall, or weight the metric toward the mistakes that cost your business more.

Compare cost after checking quality

  • Perplexity, Liquid d1-ultra and OpenAI Decisions reach the illustration’s candidate region.
  • A cost-quality plot shows when a cheaper model offers similar performance and when better performance costs more.
  • The article reports infeasible training splits for Perplexity and d1-ultra; the plotted minimums do not guarantee held-out performance.
  • These rankings describe one broad failure-finding task; narrower judges and other data can change the result.

A score needs calibration before it means confidence

  • Group similar scores and compare them with observed failure rates before treating 0.8 as an 80% risk.
  • Platt scaling learns a probability adjustment; it helps some models in this example and worsens others.
  • Binary alerting mainly needs a well-tested threshold; probability-based decisions add the calibration requirement.
  • Revisit thresholds when the prompt, task or data changes.

“Don’t just choose F1 because we did in this blog post.” — Hamel Husain, 14:00

The companion article supplies saved scores, prompts and the experiment’s methodological qualifications.