Joe Barrow (ex-Amazon/Adobe, ML lead at Pattern Data, now Adobe Research’s Document Intelligence Lab) on choosing an OCR model for AI document processing. 24 minutes on Hamel Husain’s channel.
Why OCR matters Your app sits downstream of OCR quality — garbage in, garbage out, no matter what the LLM does after It’s not solved — Anthropic shipped bad PDF handling for a year because it was pulling text, not doing OCR; users noticed OCR is sticky — once you build on a vendor, swapping models is painful (Pattern learned this the hard way) Documents are evil — multi-column layouts, rotated scans, no reading order; TeX-compiled PDFs have no spaces (glyph glue), so naive text extraction gives you one run of characters The decision grid: two axes Text blocks vs. document structure Text blocks: word/line bounding boxes → grounding, evidence highlighting, cheapest Structure: headings, reading order, grouped paragraphs, tables, figure alt text, chart de-rendering → much better LLM input (LLMs are trained on markdown-like structure; raw line runs look like garbage to them) API vs. self-host API: ease of use, vendor support (startups retrain on your bad docs), minimal time — right for ~95% of teams Self-host: control throughput/concurrency (APIs cap concurrent docs — a real bottleneck), stable weights, domain fine-tuning, no lock-in, cheaper at bulk — but only if your time ≈ $0 or you run huge batches The four quadrants Big cloud APIs (AWS Textract, Google Cloud Vision, Azure) — $0.60–1.50 / 1k pages; word+line boxes only; tables/forms a la carte at $10–15 / 1k Document startups (Reducto, Data Lab, Extend, LlamaIndex) — $5–20 / 1k pages, “fast” vs “accurate” tiers; structure included (markdown/HTML, tables, figure boxes) Open pipelines (PaddleOCR, Nemo Tron, Tesseract) — 10–100M params, nearly free, edge-deployable (PaddleOCR runs on phones/e-ink); text lines only, post-process with layout models Open VLMs (LightOn OCR 2, GLM OCR, GOT-OCR, Chandra/Surya) — 600M–8B params, native document structure, ~$0.20–0.30 / 1k pages on a saturated H100; hallucination risk exists but clouds hallucinate on crusty scans too How to actually choose Ignore benchmarks (OmniDocBench, CR Bench) — they’re not run on your data Build a 50–100 page sample of your own representative PDFs Run a few candidates, diff the returned text (catches junk-on-handwriting fast), visualize the boxes ~a day of effort total — then pick Watch the license Chandra/Surya (Data Lab): free only if org < $2M revenue AND not competing with Data Lab LightOn OCR: Apache. GLM OCR: MIT (but relies on PaddlePaddle’s Doc Layout model — Apache — both apply) Self-hosting, for the ~5% who should Inference engines: VL (default, OpenAI-style client) or SGLang; infra: Modal (request-queue scaling beats SageMaker), BaseTen, or big cloud for one-off batches 1B-param models (LightOn, GLM) on H100 → ~10k pages/hr, 20–30¢ / 1k pages; 4×3090 ≈ one H100 → 3–4 pages/sec His 7M-page local-laws dataset: ran over a weekend at ~30¢ / 1k all-in “You can process 1,000 pages per second, but it doesn’t matter if they’re all wrong — then your entire app’s output is going to be garbage.”
...