Sofia Jakovcevic, a forward deployed engineer at Arize AI, demonstrates agent regression testing in this 19-minute AI Engineer talk.

One small edit can change the whole workflow

  • In her support-agent example, adding confirmation before every action improves the safety score by 17 points and reduces task completion by 17 points.
  • Extra confirmation steps leave work unfinished when users abandon the session.
  • Averaging those dimensions hides the trade-off. Keep safety and completion visible separately.
  • A trace, the recorded sequence of an agent’s steps, reveals changed tool calls, handoffs and behavior that a prompt diff cannot show.

Build the five-step regression loop

  1. Curate a dataset: include typical production requests, known failures, edge cases and successful workflows worth protecting.
  2. Choose evaluators: use code for deterministic checks and judge models, which grade outputs against a rubric, for qualitative behavior.
  3. Run a baseline: record the current agent’s outputs and scores on that dataset.
  4. Compare changes: rerun prompt, model, tool or handoff changes and inspect each metric.
  5. Gate merges: run the checks in continuous integration, the automated tests attached to a code change, before accepting it.

Test the prompt and the complete agent

  • Her Arize demo uses a curated 100-row dataset and compares prompt versions with action-safety and task-completion evaluators.
  • The prompt playground tests one configured prompt with tools; the agent playground calls a remote agent to exercise its broader workflow.
  • Track latency, cost and error rates alongside behavioral scores.
  • A model upgrade needs evidence of value on the relevant workload, especially when it costs more.

Turn failures into lasting tests

  • Add newly discovered production failures to the regression suite, so fixes remain covered.
  • Connect coding agents to traces and experiment results through tools or skills, reusable instructions for working with those systems.
  • Jakovcevic proposes agents that investigate failures, suggest fixes and rerun evaluations using the same feedback loop.
  • Judge quality, representative cases and release criteria still determine what a passing score means. Automated improvement is her direction of travel, not a demonstrated safety guarantee.

Caveat: the score changes come from a presented support-agent example and vendor demo. They do not establish a general effect size or justify weakening required approvals.

“Every failure should become a test.” — Sofia Jakovcevic, 17:14–17:16