Nathanael Weill explains answer-first testing for specialized business agents in this 20-minute AI Engineer talk.

Make the test cheaper than the change

  • His earlier telecom assistant answered from documents retrieved at question time, a method called RAG.
  • Weekly expert panels limited experiments to roughly one a week, while extra pipeline complexity introduced regressions.
  • Generating questions from known source pages let code check retrieval and enabled dozens of evaluations a day.

Five ways to construct known answers

  1. Compare original and approved contracts; this failed because the team lacked the original versions.
  2. Start from an explicit rule and plant a violation in generated text; Weill reports 275 tests.
  3. Remove clauses from approved contracts and check whether the agent detects the omissions.
  4. Change irrelevant details or clause order and check that the result stays consistent.
  5. Fill expert-authored templates with combinations whose expected flags are known.

Prefer checks you can explain

  • The contract workflow spots clauses, retrieves support, drafts amendments and decides whether to keep them.
  • Code-based checks avoid adding a second uncertain model when the answer is already known.
  • Clause-removal tests exposed many misses; passing easier tests would have hidden that weakness.
  • He reports reducing a four-week feedback cycle to a roughly five-minute evaluation run in this project.

Keep experts in the loop

  • Use real expert feedback to improve the test generator and coverage, rather than patching individual examples only.
  • Synthetic tests are a development tool; they do not establish real-world coverage or legal readiness.
  • Sometimes the agent’s design must change to make its behavior testable.

“If you cannot grade the agent, you can’t make it better.” — Weill, 07:30