Turn Agent Failures Into Tested Fixes — Fuad Ali

Fuad Ali, a product manager at Arize AI, walks through agent debugging and testing in this fifty-three-minute AI Engineer workshop.

Start with evidence the next agent can use

A trace records a request’s model calls, tool arguments, results and timing; group related requests into sessions.
Arize Signal reads a project’s traces on a schedule, clusters recurring problems and can inspect the connected GitHub repository.
Its default six-hour schedule is configurable; an alert can also trigger an investigation.
Keep the issue, supporting traces, test cases, scores and human annotations connected for the next run.

Wonder Toys stores product categories as lowercase, hyphenated names; its model supplies title-cased names with spaces.
The mismatch returns empty searches, causing broad fallback searches, repeated model calls and slow answers.
Signal links the symptoms to source code and drafts a pull request that normalizes categories.
Ali deliberately planted this bug; the demonstrated repair is a controlled example.

Save meaningful failures as regression cases, examples that future changes must continue to handle.
Deduplicate recurring findings so one defect creates one issue rather than a pile of alerts.
Have a person decide which findings belong in the dataset and what a successful result should look like.
Recheck older cases as the dataset grows; fixing one prompt or tool can change behavior elsewhere.

Use code for exact rules: valid JSON, required fields, normalized filters and tool arguments.
Use a judge model, a model grading another model’s output, for judgments that need interpretation.
An agent can investigate a broader goal with several calls; that adds cost and leaves the success criteria needing scrutiny.
Ali discusses cheaper decision models and confidence-based review queues, while warning that thresholds depend on the use case.
His draft retry check flags repeated product searches; legitimate multi-search requests need to be represented when refining it.

Run the revised agent on saved failures and compare retry behavior, latency and the other checks that must still pass.
Inspect individual outputs and annotate disagreements; a passing score only covers the chosen cases and grading rules.
Trace the reviewer and its evaluators too, but stop adding layers when customer ground truth runs out.
Keep issue confirmation, test design, dataset curation, code review and release approval as explicit human decisions.

Production sampling, model choice, review frequency and context size all affect the cost of finding and checking a fix.
Ali’s $8 agent versus $20 improvement-loop example is hypothetical, not a measured bill or recommended spending ratio.
His toy agent can restart with its own database; testing systems with external dependencies remains a harder setup problem.
Caveat: the title promises self-improvement while you sleep. The workshop shows a vendor-assisted repair workflow with manual steps; near-zero-human operation remains a stated ambition.

“You can’t just trust every single issue that comes out of this agent.” — Fuad Ali, 32:03–32:08