Dermatologist Magnus Lynch tested two image-capable language models against Inception v3, a specialized image classifier, on 200 biopsy-confirmed skin lesions. The classifier’s architecture dates to 2015, but Lynch fine-tuned it recently on 68,011 dermatology images. He reports excluding all test patients from training and validation.

  • Naming the condition: the classifier got 125 of 200 first-choice diagnoses right, versus 84 for GPT-6 Astra and 85 for Claude Fable 5.1.
  • Deciding who can go home: at its preset threshold, the classifier discharged 18 of 30 melanomas. The language models discharged three and four.
  • Changing that threshold changes the result: Lynch’s table reports one or two melanomas discharged under matched conditions. Those are separate settings from the preset classifier.

Overall accuracy and safe discharge are different tests. This small, author-run comparison shows why both belong in a model evaluation; it offers no basis for using the demonstration to diagnose yourself. Each language-model case was run once, and every lesion was suspicious enough to have been biopsied.

The discussion on Hacker News adds questions about how to build and test such systems.

What the thread adds

  • EagnaIonat — pushes back on Legend2440’s claim that specialized training is benchmark gaming: “The only way to determine if it’s brittle is to test on a different dataset and look at the sensitivity and specificity.” That asks how many dangerous cases it catches and harmless cases it clears on fresh data.
  • snitty — proposes a different role for language models: “Just use the LLM to make a thing that’s actually good at what you need it to do.”
  • jonex — asks what the image-only comparison leaves open: “What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those?” The article does not test those conditions.
  • vikramkr — questions what the discharge test measures: “Are you evaluating the model on if it can replace the need for a biopsy itself”. Lynch confirms that every lesion was biopsied and scores against the tissue diagnosis. The commenter asks whether that measures hindsight about benign lesions or the earlier decision to investigate them. This challenges the interpretation of the discharge results, while leaving the diagnosis counts intact.
  • bunderbunder — qualifies the advice to train a specialist: “It’s basically choosing lower model development costs in exchange for higher inference costs”. They argue that scarce labels and low usage can favor a prompted classifier. That is an economic argument for lower-stakes applications, separate from evidence about medical safety.

Handles are pseudonymous; HN publishes no per-comment scores. The fetched slice follows HN’s ranking, and these selections are not a consensus.