Noyan Tokgozoglu, head of AI Engineering at Optiver, describes the trading firm’s internal AI platform in this 23-minute AI Engineer talk.

Make new models measurable before rollout

  • Optiver aims to make newly released frontier models usable in production workflows the same day, after testing their value.
  • Trading teams supply “golden sets”: internal examples used to compare models on the firm’s actual work.
  • Adoption also needs gateway integration, compatible APIs, tool-running software updates, budgets and user guidance.
  • Regional support rotations keep the global service operating after launch.

Separate the platform’s layers

  • An inference gateway routes requests across external providers and self-hosted models, tracks spending and supports provider failover.
  • An agent gateway registers agents and coordinates their identities and interactions.
  • A “meta harness” shares team configuration across different harnesses, the software that runs agents’ tools and tasks.
  • Infrastructure runs on-premises, with external model inference as an explicit exception under data-retention agreements.

Cut the context bill before chasing clever routing

  • Some fresh sessions started with roughly 200,000 tokens of context because teams loaded too many tools and instructions.
  • Tool search, command-line wrappers and better cache settings reduced that starting load.
  • Moving knowledge out of oversized prompts lets agents retrieve what a task needs.
  • Team-visible spending, alerts and self-set limits helped users manage consumption.
  • Smaller models and splitting planning from execution offer further options when task quality holds up.

Build around how people already work

  • Optiver consolidated bespoke setups into a portal for sessions, chat, artifacts and team configurations.
  • Traders contribute improvements, including collaborative document editing; shared rooms for agents are another direction of development.
  • Narrow, versioned agents keep capabilities easier to test than one agent that absorbs every team’s tools.

Replay remains an open problem

  • Logging each tool call supplies evaluation data, but a later run may see a different world.
  • Point-in-time trading databases are easier to reconstruct than a historical Slack workspace.
  • Optiver is exploring snapshots, tool proxies and confidence estimates for how faithfully an evaluation reproduces earlier conditions.
  • Caveat: this is an internal architecture report. It supplies neither quantified savings nor a controlled estimate of AI’s trading-profit contribution.

“We move at the speed that they let us move.” — Tokgozoglu on relying too heavily on one provider’s roadmap, 18:10