Gbenga Awodokun shares a LangGraph medical-scribe tutorial with interchangeable OpenAI and Jev routing steps. Its recorded benchmark makes a narrow comparison inspectable: which transcript should reach a specialty writer, and which should stop for review?

The measured step comes before writing

  • Jev answers three questions together: specialty, whether the input needs review, and urgency.
  • Ordinary code sends low-confidence or unsuitable input to a review branch. That branch records a message and stops; a clinician queue remains future application work.
  • A generative model still writes the clinical note. The benchmark runs only the routing step and never grades a finished note.

The article describes GPT-5.6 Terra grading note sections. The public benchmark instead compares gpt-4o-mini with Jev on transcript routing. Those are different tasks and baselines.

Equal scores conceal different mistakes

The author’s saved results cover six fictional transcripts, five runs each per backend. Both backends match the expected route in 26 of 30 runs, reported as 87% accuracy and 97% consistency.

  • On a two-line fragment, the LLM routes to a writer in four of five runs, despite reporting 0.90 confidence. Jev sends all five for review.
  • On a 17-year-old’s visit, Jev’s specialty confidence averages 0.54. The router abstains in four runs, which the benchmark scores as errors because its expected route is pediatrics.
  • Choosing between safe abstention and automatic routing needs its own policy. Aggregate accuracy prices both kinds of disagreement equally.

Typed output is not a correct decision. This tiny test also cannot establish calibrated confidence, and the recorded variation contradicts a blanket claim of deterministic decisions.

Price the whole workflow

The author’s saved summary reports 1.17 seconds versus 0.13 seconds median routing time, and $0.203 versus $0.045 per 1,000 routing calls. Offline arithmetic using the unrounded results gives about 8.9× faster and 4.5× cheaper for this step, not a reproduced live benchmark.

The README separately estimates $0.00066 versus $0.00054 per complete short-sample run, including note generation. Those author-reported token-cost estimates exclude hosting, human review, and clinical validation. The routing multiplier does not describe total savings.

Keep the safety boundary visible

The code checks review probability above 0.5 and specialty confidence below 0.6. Its urgency score is logged but never controls the route. Jev API errors of the handled SDK type stop automatic writing.

Before a clinical deployment, evaluate note omissions and factual errors separately, validate escalation thresholds, and account for transcripts sent to hosted models and tracing services. The repository explicitly labels its data fictional and its workflow an unvalidated teaching example.

Caveat: source and code inspected; all 60 saved decisions replayed through the router and summary arithmetic checked offline. No live model calls, timing reproduction, clinical validation, or security audit.