Ashwin Paranjape, founding AI lead at Samaya AI, presents his team’s financial-agent experiment at AI Engineer.

A more measurable forecasting task

  • Stock prices are noisy and difficult to explain; quarterly earnings provide reported figures, analyst consensus and management commentary to compare with predictions.
  • The agent uses company documents, prices, consensus and timestamped news through a harness that restricts data to the chosen prediction time. Web access is replaced by timestamped news because arbitrary web pages are difficult to time-gate.

What Samaya reports

  • Across more than 450 earnings reports and seven models, Paranjape says newer models improved on several measures; GPT-6 Astra beat even bias-corrected consensus on revenue prediction error in this test.
  • The team also compared direction-of-surprise hit rates and the correlation between predicted and actual surprise magnitudes. These are company-reported results, not an independent replication or evidence of profitable trading.
  • In one example, an agent used a supplier’s information to forecast a semiconductor company’s results; in another, it found evidence of a tariff refund but did not incorporate it into its forecast.

Data and guidance

  • The team’s comparisons suggest current data matters: model knowledge alone performed poorly, and an 11-week-old data cutoff reduced performance relative to timely information.
  • Expert guidance led to more research and improved prediction error for one model. The strongest model showed no statistically significant difference with consensus withheld in this setup.

“Can we forecast the future? Can we forecast it better than what we do today?”