Yikai Zhao and colleagues test whether document conversion preserves the requirements an author marked for attention. Words can survive extraction while color and highlighting disappear.
Their task asks models to recover specially marked requirements from real policy documents. It measures a specific loss in the material reaching the assistant.
- On 76 documents, the authors’ formatting-aware representation scored 0.690 micro F1, which balances retrieving marked requirements against listing the wrong ones.
- Docling scored 0.150, plain text 0.075 and formatting-preserving docx2python 0.586.
- Removing highlighting or color caused the largest losses. Bold, italic, monospace and strikethrough each changed the measured score by at most 0.003.
- Preserving cues helped across six answering models. Adding interpreted role labels helped less consistently and reached significance on three.
The parser recovers cues without model requests, but the representation costs roughly twice the tokens of plain text. Its PDF evaluation ties the strongest baseline overall on cue recovery; scanned pages still need separate support.
Caveat: The task explicitly depends on author markings. Its markup-derived labels await human validation, and the results do not show that formatting improves every document task.