Will Brown, who leads applied research at Prime Intellect, describes a proposed path from agent workflows to organization-specific model training at AI Engineer.

The knowledge gap

  • Companies accumulate preferences, decision patterns and informal processes alongside their documented rules.
  • Brown argues that agents repeatedly rediscover this context, consuming tokens and missing knowledge held by experienced employees.
  • Skill files and retrieval supply context; training changes model weights. He suggests reserving training for recurring patterns that earn the investment.

Start with tests and routing

  • Collect agent traces, prompts and feedback, then turn representative examples into evals: tests of model behavior.
  • Compare a smaller model with a larger one on a task and route suitable cases to the cheaper model, with human or agent feedback.
  • Decision models could learn repeated human preferences, but the talk does not establish that these decisions are safe to automate without oversight.

Turn decisions into training signals

  • Brown proposes small, dynamically generated interface cards that ask people for a specific decision instead of burying questions in agent threads.
  • The answers can provide labels for training; the quality of those labels and the surrounding review still matter.
  • Prime Intellect is experimenting with converting production traces into training environments combining real and synthetic examples, with human oversight.

A longer-term loop

  • Brown describes an internal prototype that surfaces flagged issues and proposed experiments for possible overnight training runs.
  • He cites another company’s roughly $300,000 model-training example for spreadsheet and slide work; its performance and full ownership costs are not independently established in this talk.
  • Caveat: This is a vendor’s roadmap and set of examples, not a controlled comparison showing that every company should train a model. Privacy, evaluation, maintenance and workload economics need separate assessment.

“Things don’t just go in the weights right away.”