Fuad Ali, a product manager at Arize AI, reviews the engineering decisions behind Alex, its in-product agent, in this 20:06 AI Engineer talk.

Let the harness own the plan

  • The harness is the software around the model that manages execution, state and tool calls.
  • Alex’s model-managed to-do list generated excessive bookkeeping and lost focus over long sequences.
  • Arize moved plan state into the harness and rendered it alongside instructions on each model call.
  • The team is testing removal of the to-do tool entirely; that final simplification remains an experiment.

Make completion explicit

  • Some models announce an action before calling a tool, so a text-only response can end a task prematurely.
  • Alex uses an explicit finish call, which can be rejected up to three times while plan items remain open.
  • A warning precedes the iteration limit, followed by a forced finish to bound execution.

Keep context and memory small

  • Large trace results become files with pointers and smaller previews; the agent can inspect details with code.
  • Checkpoints replace older execution history within a fixed token budget.
  • Long-term memory is one Markdown file per user, capped at 8,000 characters and updated after the session.
  • Ali says that file beat the team’s more elaborate memory designs; the talk supplies no comparative benchmark.

Simplify the agent, then test the changes

  • Arize consolidated 10 specialists into one agent that loads skills and tools on demand.
  • The team prefers a fast, adequate default model and spends effort on the harness.
  • Model routing and narrow post-trained models are being explored; simple filter generation can use one model call.
  • Production sessions become regression examples, including multi-turn sessions and specific turns for replay.
  • Traces expose failures hidden behind successful status codes, such as 43 identical tool calls in one turn.

Preserve human judgment

  • People supply reference labels, check model-based evaluators and choose acceptable speed–quality trade-offs.
  • A proposed fix needs regression testing because it can create the next bug.
  • Code performs arithmetic and aggregation; the model interprets and explains the result.
  • Ali’s account is a vendor production retrospective; the illustrative failure counts do not establish population-wide reliability.

“The loop, the plan, the stopping rule, and the memory are all ours.” — Fuad Ali, 06:07