Louis-François Bouchard of Towards AI presents Tone-Bench in this 19-minute AI Engineer talk.
Slop has more than one symptom
- Overused vocabulary, vague claims, empty content, repetition, invented facts and an interchangeable voice all contribute.
- Tone-Bench asks whether a model can write a video script in Bouchard’s voice, given research, instructions and examples.
- Its rankings describe one editorial target; another writer needs their own tests.
Test consistency and the graders
- The reported setup uses 10 script tasks, five drafts per model, four model-based graders and nine scoring metrics.
- Categories cover voice, writing quality, instruction following, research accuracy and slop.
- Repeated drafts expose variability that a single impressive output can hide.
- Bouchard reports model-family preferences among graders; his team also compared 1,000 human A/B excerpt pairs.
Good rubrics cost real work
- He reports roughly 500 kickoff hours for scripts, scoring rules and examples, plus months of iteration.
- Regrading after a rubric change costs over $2,000 in his setup; changing generation prompts costs over $4,000.
- Those figures describe rerunning his benchmark, not the price of writing one article.
Writing workflow
- Give the writer concrete positive and negative examples, research and explicit voice guidance.
- Try a limited review pass from a strong second model; repeated review can drift back into generic writing.
- He proposes generating several cheap drafts, then selecting and combining the strongest parts.
- Human editing remains necessary; he explicitly lacks quantitative evidence for some workflow recommendations.
“You just want to find a model that will be the least annoying to you to get to where you want to be.” — Bouchard, 15:51