Datalab founder Vik Paruchuri joins Hamel Husain for a 31-minute talk and Q&A on measuring optical character recognition (OCR) and document parsing.

A leaderboard can reward the wrong improvement

  • Paruchuri says 70 reinforcement-learning steps moved one model from 23rd to third on ParsBench without improving its OCR.
  • He reports another extraction score moving from 64 to 93 after scoring fixes; both are his accounts, not independently reproduced results.
  • A wrong financial digit can barely change text similarity while corrupting every downstream answer.
  • Conversely, correct HTML and Markdown can look very different to a character-edit-distance metric.

Audit the test before trusting the score

  • Good metrics accept correct outputs, reject wrong ones, explain failures, have traceable ground truth, and separate stronger from weaker systems.
  • Check whether harmless format changes alter scores and whether deliberately corrupted digits fail.
  • Inspect tests that every strong model fails; the expected answer or scoring code may be wrong.
  • Layout grouping, chart inference, and exact equation strings can reward agreement with an evaluator’s conventions.

Write checks around the document’s meaning

  • Test whether text is present, one section precedes another, or a table value belongs under the right header.
  • Normalize equivalent representations consistently across models while preserving differences your application needs.
  • Source answers from human review, publisher data, or documents whose contents you control.
  • Human review still matters: even unit-style tests can mishandle capitalization, line breaks, or equivalent mathematical notation.

Evaluate your own pipeline

  1. Sample production document classes and difficult edge cases.
  2. Write questions with independently verifiable answers and checks for costly failure modes.
  3. Compare parser outputs through the downstream task, then inspect individual failures.
  4. Report representative and hard-case results separately; easy filings left several parsers close in his example.

Caveat: Paruchuri sells a competing document-processing system. His benchmark critiques and harder-document results are attributed examples; the talk does not establish a universal best parser.

“Human verification is really critical, easily skipped these days, but really, really important.” — Vik Paruchuri, caption disfluencies trimmed