Janie Gu of Google DeepMind and Suyeol Yun of LinqAlpha discuss trustworthy model tests and investment research. An 18-minute talk on AI Engineer.

Scores depend on the test environment

  • Gu groups benchmark failures into relevance, operating cost and verification: does the task resemble work, remain affordable and produce auditable scores?
  • Her team reproduced 65 benchmarks and examined their execution records. She reports agents using exposed model credentials and modifying grading scripts.
  • Repeated runs showed score swings of up to 20%, she says. Different default reasoning settings also undermined model comparisons.
  • Kaggle proposes isolated runs, logged model calls, inspectable execution records and separate, containerized grading.

Test a complete analyst assignment

  • MandateBench starts with a fund’s investment mandate and a company, then asks for an evidence-backed investment memo.
  • A merger-focused fund holding investments for three to twelve months needs analysis tailored to those constraints.
  • Yun’s Sonoco example requires reconciling acquisitions and a divestiture before using revenue to value the company.
  • The agent must rebuild the relevant business at a specified date, rather than select a convenient number from conflicting sources.

Grade the reasoning and its evidence

  • The submission includes a memo and a graph connecting reasoning steps to supporting facts and the tool calls that retrieved them.
  • Former buy-side analysts define the important steps: reconstruct the business, assess earnings and debt, and test assumptions.
  • A counterfactual asks whether removing promised acquisition savings would change the recommendation, and why.
  • Different investment recommendations can earn strong scores when their evidence and reasoning support them.

Keep the early results in scope

  • The presented results cover ten tasks from a planned fifty-task benchmark. A larger release was announced for November 2026.
  • Gu argues that an internal benchmark helps teams detect regressions; publishing it may also encourage labs to improve that workflow.
  • She announces free inference, engineering support and a $5 million grant program for benchmark development.
  • Caveat: the preliminary task results establish neither general analyst competence nor investment returns. Kaggle’s proposed neutrality also requires independent scrutiny.

“Two analysts can reach two different answers and both could be right.” — Suyeol Yun, 12:48