Aparna Dhinakaran of Arize AI uses its Signal product to explain the changing AI-engineering role in this fifteen-minute AI Engineer talk.

Build the improvement loop

  • Dhinakaran argues that engineers increasingly design systems that build, evaluate, ship, monitor and improve AI features.
  • Signal reads traces, records of model and tool activity, to identify recurring problems and propose fixes or pull requests when given repository access.
  • Telemetry becomes input for agents investigating failures, as well as evidence for people debugging them.

Make every run bounded and observable

  • Start agents on a schedule, a user report or a metric-based event; keep the trigger separate from the work it starts.
  • Assemble relevant traces, application logs, repository access and required credentials inside a fresh sandbox.
  • Dhinakaran describes a single proxy for sandbox traffic and tracing of the investigating agent’s own actions.
  • Preserve issue history and run state outside the sandbox. Tear down the sandbox, credentials and repository access afterward.
  • External state lets a replacement sandbox or harness resume interrupted work; exact recovery semantics need engineering beyond this high-level sketch.

Test the judges as well as the agent

  • Evals test how the system behaves, including whether proposed changes preserve successful behavior.
  • Engineers should inspect traces, identify common errors and define what a good result means with human-reviewed examples.
  • Compare automated evaluators with those human judgments before shipping them; repeat the checks as models, data and tasks change.
  • Harness and model choice also need evaluation against issue quality, proposed-fix quality and cost.

Feed production evidence back into development

  • Test changes before release, evaluate production traces afterward and use the new failure evidence to guide the next investigation.
  • Trace coding agents that build the product alongside the agents users interact with.
  • Caveat: this is a vendor’s architecture account, without measured false-positive rates, repair-success rates or proof that generated pull requests are safe to merge automatically.

“We actually evaluate the evaluators.” — Aparna Dhinakaran, 12:02–12:06