SallyAnn DeLucia, Arize AI’s head of product, builds a first evaluator for a travel-planning agent in this thirty-five-minute AI Engineer workshop.

Read real runs before writing the test

  • Evals test how an AI system behaves; traces record its model calls, tool activity and intermediate results.
  • Inspect the user’s request, the steps taken and the final answer before deciding what successful behavior means.
  • In the demo, a user wants three days in Prague within an $800 budget, with a more local, less touristy itinerary.
  • Use the root span, the top-level record, to judge the whole answer; select a tool span when judging that tool’s work.

Turn “helpful” into a checkable criterion

  • Specify the task, what good and bad look like, the evidence to judge and explicit pass/fail conditions.
  • Test one quality at a time: follow-up relevance, requested trip length or consistency with the stated budget.
  • Start small and expand with real failures. DeLucia says Arize’s Alex agent grew from a few checks to 32 evaluators.
  • Use ordinary code for deterministic checks such as valid JSON; use a model-based judge where the criterion needs interpretation.

Build and inspect the judge

  1. Filter the relevant spans and save them as a small dataset.
  2. Turn the criterion into a prompt with the judge’s task, required evidence and explicit label rules.
  3. Map prompt variables to the dataset fields; choose labels such as correct, incorrect or not applicable.
  4. Run the judge, read its explanations and inspect disagreements before revising the prompt.
  5. Save the evaluator for reuse in development and on production traces.

Check the evaluator against people

  • Have people label examples, then compare the automated judge’s decisions with those judgments.
  • Involve product, engineering, support, domain experts and security in defining the outcomes that matter.
  • Keep business metrics alongside targeted evals: changes in outcomes and specific failure labels answer different questions.
  • Caveat: this is a vendor-led workflow demonstration. It provides no measured judge–human agreement or evidence that the demo evaluator is ready to gate releases.

“Start small, 20 traces maybe, one dimension, one judge.” — SallyAnn DeLucia, 34:28–34:32