Together AI’s Hassan El Mghari and his father Omar, a former CFO and internal-audit head, present a Costco forecasting experiment on AI Engineer.
Build a forecast without leaking the answer
- They chose a cutoff before Costco’s July 2024 membership-price announcement and tested whether candidate models already knew about it.
- Their pipeline extracted roughly 3,000 timestamped facts from 214 public filings and filtered them to the cutoff.
- An open-weight model generated structured assumptions and 14 formulas; a separate spreadsheet performed the calculations. Membership fees, for example, had to be recognized over 52 weeks.
- Sixteen runs cost a reported 19 cents for that step. Basic formula checks and up to two repair passes left five builds that passed.
Audit the assumptions, not just the output
- The five selected builds’ forecast error averaged 4.4% against the reported outcome, according to Hassan. Passing their engineering checks did not validate the accounting assumptions.
- Omar found that the model set executive-member share at 15% despite a disclosed figure of 45%. Its cited sentence was real, but did not support the invented percentage.
- It also applied the US-and-Canada fee increase to members outside those markets. The overstated population and understated executive share partly canceled each other.
- Correcting only the executive share made one illustrated error worse. Omar’s point: a close forecast can conceal a broken mechanism.
Put domain judgment into the checks
- Add tests for cited numbers, geographic scope and assumptions that should change the result when varied.
- Let a subject-matter expert review the model before promoting repeatable checks into code. Omar maps this to management controls, internal audit and independent review.
- Caveat: this is a creator-presented case study on one company and event, not an independently reproduced forecasting benchmark.
“Luck is not a control in the auditing world.” — Omar El Mghari