Stephanie Jarmak of Omni Analytics brings an astronomer’s experimental discipline to agent testing in this 17-minute AI Engineer talk.
Decide what the test must answer
- A prompt change made one run 28% cheaper, but the agent still ignored the search tool she wanted to test.
- Public benchmarks, fixed tasks for comparing systems, can help shortlist candidates; your own workload should decide deployment.
- Test the model together with its harness, the software that supplies tools and runs the task.
Build an experiment around real work
- Sample task families from production: understanding code, fixing bugs, answering questions or building dashboards.
- Repeat runs to expose variation and compare changes under the configuration you will actually ship.
- Retain traces: was the tool available, was it called, what came back, and did the answer use it?
- Match the score to the goal: retrieving 295 files when six matter can look perfect on recall while wasting context.
Make safeguards operational
- A warning needs a code-enforced stop, an escalation path and a reviewer who can act.
- Even a small false-alarm rate can overwhelm expert reviewers at high volume.
- Keep independent records when the system being evaluated could alter its own evidence.
- Check production outcomes after offline tests; use shadow runs when changes carry substantial risk.
“Production is truth.” — Stephanie Jarmak, 14:38