Hamel Husain and Isaac Flath run a model’s writing through the cheapest eval there is: read the drafts out loud and argue about them. (Two-minute clip from the same “Writing Evals With Isaac” live session.)

The task

  • Isaac’s blog post describes a document QA pipeline built over a 77-page set with 22 questions. Every answer cites a bounding box drawn on the exact spot on the page, so checking an answer takes seconds instead of a full re-read
  • They hand the post to a model and ask for an X post about it, focused on the pieces of the pipeline that can be tested
  • Three drafts, one task, different prompts — same three candidates as the session’s writing-style comparison

What they liked

  • A real number, early and concrete: the model answers 197 for the underwriting fee — and “if you have to reread the whole document to find out, the answer saved you nothing”
  • The pipeline stated as testable stages with the reason for splitting it: once checking is cheap, experiments are cheap, so you can swap one stage at a time and rerun the same 22 questions
  • One fewer fact. The winning drafts carried less than the others, and that was an improvement rather than a loss
  • Precision in naming: “finding (grep vs semantic search), cheapest run” beats “GPT vs semantic search over the same OCR output, same accuracy, cheapest run” — the longer version buries the point that the bottleneck was never search

What they didn’t

  • “Very similar but a little bit more wordy” is the entire verdict on one draft
  • Drafts that pull the pipeline’s three steps out of the source post without explaining what they are — accurate and unreadable
  • An extra fact about cost that added nothing to the argument
  • Editorial framing that tells the reader what the result means before showing it

The rule the description writes down

An AI draft can get the facts right and still not be good. The ones we liked had a real number in them and one less fact — turns out cutting helped more than adding.

Why this counts as an eval

  • The criteria came out of reading the outputs, not from a rubric written in advance — which is the sequence the session is built around: look at the drafts first, then name what is wrong with them
  • Bounding-box citations are what make this style of evaluation possible in the first place: cheap checking turns “trust me” into “check it in five seconds,” and cheap checking is what makes comparing whole pipelines practical instead of eyeballing one