Khawaja Shams of Momento joins Hamel Husain for a 24-minute talk about building tests for coding agents, models that use tools to change software.

Start with tasks that resemble your work

  • Public benchmarks, fixed tasks for comparing models, can miss your languages, frameworks, coding style, and definition of done.
  • Replay historical bug fixes from your own pull requests: restore the earlier code and check whether the agent repairs the failing test.
  • Combine measurable tasks with representative ones. A neatly isolated bug gives a clear score; everyday development brings ambiguity.
  • Vary prompts and session history. Naming the exact file, test, and grading command removes work a developer normally delegates.

Test the whole setup

  • The prompt, harness, model, reasoning setting, provider, and policy all affect the result.
  • The harness is the software around the model that manages its tools, files, and commands.
  • Long or conflicting context and many tool calls can expose weaknesses that difficult programming puzzles miss.
  • Use automated tests where possible; check a judge model’s grades against human judgments when tasks need qualitative review.

Keep completed answers out of reach

  • One agent recovered a historical fix by moving the repository forward through Git history.
  • Another found a previous agent’s completed answer on the local filesystem.
  • Use point-in-time snapshots, fresh session memory, restricted network access, and recorded tool calls.
  • Live search and company knowledge systems add changing inputs. Record those dependencies and inspect the resulting traces.

Repeat runs and investigate failures

  • Shams describes a cherry-picked comparison that appeared 6.6 times better; the aggregate difference was about 16%.
  • Provider throttling, gateway errors, and hung calls can look like model failures until someone reads the execution trace.
  • Reliable automated checking makes retries practical. Ambiguous outcomes demand more consistent performance and stronger review.
  • Track speed, correctness, cost, and policy adherence separately.

Separate coding time from grading time

  • A test suite taking roughly 20 minutes obscured differences between coding runs lasting less than a minute.
  • Instrument the harness to expose costs, turns, timing, errors, and the sequence of tool calls.
  • Revisit the scoring system whenever a surprising result appears; the evaluator itself needs validation.

Caveat: Shams presents lessons from a little over a month of work. These are early engineering observations, not a validated model ranking.

“A single run is not a measurement.” — Khawaja Shams, 20:05