Xander Song, an Arize AI engineer working on Phoenix, explains Harbor through a fifty-six-minute presentation, live demonstration and audience questions on AI Engineer.

Give the agent a real task and inspect the result

  • Harbor runs agents, models carrying out multi-step tasks with tools, in fresh sandboxed environments with realistic data.
  • A task bundles natural-language instructions, an environment and verifiers, scripts that check whether the intended changes happened.
  • For a broken database migration, check that the migration succeeds and leaves the expected columns and indexes.
  • Keep tasks and success criteria stable while changing prompts, tools, skills or the surrounding agent software.
  • Run several independent trials because the same agent can behave differently on each attempt.

Test the test setup before paying for model calls

  • An oracle is a known solution script that exercises the environment and verifiers before an agent enters the picture.
  • Song describes mounting tests after the agent finishes so it cannot inspect the checks during the run.
  • Passing the oracle catches setup mistakes; it still leaves the team responsible for whether the checks express the right requirements.
  • Record trajectories, the agent’s sequence of steps, for debugging alongside the final outcome.
  • Song reports using Daytona with about 200 concurrent workers; this is his team’s configuration, not a measured speedup.

Evaluate the product around the coding agent

  • Phoenix tests three interfaces: a coding agent with its command-line tool, a coding agent with its MCP tool connection, and its server-side Pixie agent.
  • Its single-node server and seeded SQLite database fit in a container, making realistic tests comparatively straightforward.
  • A task can ask the agent to find hallucinated model calls and add them to a dataset; a verifier checks the resulting records.
  • Harbor runs outside the sandbox as the orchestrator. Each trial tests one configured agent interface.
  • Harbor works independently; Phoenix adds dataset versioning, experiment comparisons and trace inspection.

A SQL tool needs a prompt and a comparison

  • Questions about total spending or the busiest day led agents to fetch many API pages and calculate aggregates themselves.
  • The team added a read-only SQL tool, then changed the prompt to favor it for counts and grouped queries.
  • Song used Claude to draft questions and reference solutions, spot-checked questions and ran the solutions against the verifiers.
  • In the demonstrated comparison, aggregate SQL use increased, GraphQL use and total tool calls fell, and the correctness reward stayed high.
  • Inspect individual tasks: an aggregate improvement can conceal a different pattern on a particular example.

Keep correctness, efficiency and coverage separate

  • Outcome checks survive changes in implementation; checks for a particular tool or skill can diagnose routing but require more maintenance.
  • The SQL example optimizes an already-working system. It supplies no independently reproduced savings figure or general model ranking.
  • A live natural-language filter fails; Song falls back to the known query expression before comparing traces.
  • Skill regression testing is early-stage, focused on important happy paths across supported models and providers.
  • Caveat: Phoenix’s compact architecture makes this setup easier. Song leaves distributed-system packaging unresolved and identifies realistic test data as the hard part.

“All they care about is what the agent leaves behind.” — Xander Song on outcome-based verifiers, 10:59–11:01