Ryan Stevens, who leads applied science at Ramp, describes how the company evaluated its Stack bookkeeping agent before launch.
Start with the job
- Work with bookkeepers to break accounting into tasks such as reconciliation, data entry, variance analysis, schedules and accruals.
- Build synthetic businesses with coherent ledgers, customer and vendor lists, bills and other data, then rerun tasks against the same initial state.
- Ramp says it spent four to six months building the evaluation before launch, with Mercor as a partner.
Measure the outcome
- Each task had roughly 10–20 success criteria; the team initially graded final outputs rather than specific tool-call sequences.
- Partial-credit accuracy and an all-criteria pass metric answer different questions: a nearly correct accounting result may still need review.
- Stevens reports that reasoning failures were prominent in Ramp’s accounting tests. This is a vendor’s result on its own benchmark, not a general ranking.
Use the evaluation repeatedly
- Skill ablation: remove or add skills and compare results to find candidates for deletion or compression; Stevens cautions that he may have reversed the signs on his slide.
- Memory tests: run related tasks across two periods, write a memory after the first, and test whether it helps in the next period.
- Regression checks: schedule recurring runs and have a background agent flag possible regressions for people to investigate.
- Automated prompt and skill tuning did not work for their setup; Stevens cautions that this may be specific to their agent.
These are synthetic, pre-launch tests; the talk does not demonstrate production accuracy or establish that final-output scoring is sufficient for safety-sensitive workflows.
“Break down a job into discrete … tasks. Build realistic environments.”