Joe Barrow (ex-Amazon/Adobe, ML lead at Pattern Data, now Adobe Research’s Document Intelligence Lab) on choosing an OCR model for AI document processing. 24 minutes on Hamel Husain’s channel.

Why OCR matters

  • Your app sits downstream of OCR quality — garbage in, garbage out, no matter what the LLM does after
  • It’s not solved — Anthropic shipped bad PDF handling for a year because it was pulling text, not doing OCR; users noticed
  • OCR is sticky — once you build on a vendor, swapping models is painful (Pattern learned this the hard way)
  • Documents are evil — multi-column layouts, rotated scans, no reading order; TeX-compiled PDFs have no spaces (glyph glue), so naive text extraction gives you one run of characters

The decision grid: two axes

  1. Text blocks vs. document structure
    • Text blocks: word/line bounding boxes → grounding, evidence highlighting, cheapest
    • Structure: headings, reading order, grouped paragraphs, tables, figure alt text, chart de-rendering → much better LLM input (LLMs are trained on markdown-like structure; raw line runs look like garbage to them)
  2. API vs. self-host
    • API: ease of use, vendor support (startups retrain on your bad docs), minimal time — right for ~95% of teams
    • Self-host: control throughput/concurrency (APIs cap concurrent docs — a real bottleneck), stable weights, domain fine-tuning, no lock-in, cheaper at bulk — but only if your time ≈ $0 or you run huge batches

The four quadrants

  • Big cloud APIs (AWS Textract, Google Cloud Vision, Azure) — $0.60–1.50 / 1k pages; word+line boxes only; tables/forms a la carte at $10–15 / 1k
  • Document startups (Reducto, Data Lab, Extend, LlamaIndex) — $5–20 / 1k pages, “fast” vs “accurate” tiers; structure included (markdown/HTML, tables, figure boxes)
  • Open pipelines (PaddleOCR, Nemo Tron, Tesseract) — 10–100M params, nearly free, edge-deployable (PaddleOCR runs on phones/e-ink); text lines only, post-process with layout models
  • Open VLMs (LightOn OCR 2, GLM OCR, GOT-OCR, Chandra/Surya) — 600M–8B params, native document structure, ~$0.20–0.30 / 1k pages on a saturated H100; hallucination risk exists but clouds hallucinate on crusty scans too

How to actually choose

  1. Ignore benchmarks (OmniDocBench, CR Bench) — they’re not run on your data
  2. Build a 50–100 page sample of your own representative PDFs
  3. Run a few candidates, diff the returned text (catches junk-on-handwriting fast), visualize the boxes
  4. ~a day of effort total — then pick

Watch the license

  • Chandra/Surya (Data Lab): free only if org < $2M revenue AND not competing with Data Lab
  • LightOn OCR: Apache. GLM OCR: MIT (but relies on PaddlePaddle’s Doc Layout model — Apache — both apply)

Self-hosting, for the ~5% who should

  • Inference engines: VL (default, OpenAI-style client) or SGLang; infra: Modal (request-queue scaling beats SageMaker), BaseTen, or big cloud for one-off batches
  • 1B-param models (LightOn, GLM) on H100 → ~10k pages/hr, 20–30¢ / 1k pages; 4×3090 ≈ one H100 → 3–4 pages/sec
  • His 7M-page local-laws dataset: ran over a weekend at ~30¢ / 1k all-in

“You can process 1,000 pages per second, but it doesn’t matter if they’re all wrong — then your entire app’s output is going to be garbage.”

Watch on YouTube — full summary in the vault note.