Thomas Hikaru Clark, Matias B. Martin and Kyle Mahowald examine why AI prose can feel difficult despite being grammatically valid. Their study finds features associated with reading difficulty across model-generated conversations, coding explanations and essays. A separate experiment shows how rewarding brevity can remove useful grammatical cues.
The corpus covers fourteen models: 120 chat conversations, 45 coding sessions and 980 essays. Human comparisons include subtitles, StackExchange and 2,934 student essays. The researchers also translate model sentences through Japanese and back to compare alternative wording.
- The nine frontier and near-frontier models favor unusual word combinations and inanimate subjects with normally animate verbs. Direct human comparisons for the latter come from essays; chat and coding use translated paraphrases.
- Long grammatical dependencies and stacked nouns often decline after translation in chat and coding. Essay results are less consistent.
- Opus 5.5 scores lower than Opus 5 on nineteen of 21 difficulty-linked comparisons. Eight have non-overlapping 95% uncertainty intervals.
The training experiment uses Qwen2.5-1.5B-Instruct to turn structured events into valid English. Both “the book the professor wrote” and “the book that the professor wrote” can be correct. A penalty per output token rewards the shorter option, while a separate constraint can limit departure from the starting model.
- Across 24 runs, the researchers vary four length penalties, two strengths of that constraint and three random seeds.
- Without the constraint, every positive length penalty produces over 98% short-form choices in both trained “that” constructions.
- Preferences transfer to unseen constructions involving “that.” Broader transfer appears in only one of four other grammatical choices, and only without the constraint.
Caveat: The study measures linguistic proxies, not reader comprehension, and the small-model intervention does not establish what caused frontier models’ writing styles. At zero length penalty, unconstrained seeds already ranged from 0% to 100% short-form preference.
Brevity and clarity need separate tests. A useful word can lengthen a sentence while reducing the work a reader must do to understand it.