Fadel M. Megahed and Ying-Ju (Tessa) Chen test decision models on manufacturing defects. Their ongoing study compares hosted Jev, local Clef-Flash and CLIP, an image-and-text model used here to compare images with labeled examples.

The repository includes image-level scores, calibration code, frozen descriptions and a record of changes to the experimental design.

Count all the labels behind the prompt

The main Jev configuration receives two reference images per class with each inspection. Those four examples come from a larger labeled learning set that also guides description tuning.

  • The learning set contains 50 images per class, except metallic pans, which have 10.
  • A separate set calibrates the scores: usually 25 images per class, with 17 for pipe staples.
  • Pans have no separate calibration set. Each method reuses learning images under its specified procedure.
  • Jev, Clef-Flash and the CLIP baseline receive the same total labeled-image budget.

A small prompt can still depend on substantial labeled development data. This comparison makes that cost visible.

The winner changes with the material

The authors report these test balanced accuracies, the average of the defect-detection and clean-image success rates:

  • Microstructure: Jev 90.2%, Clef-Flash 86.7%, CLIP 93.3%.
  • Simulated textiles: Jev 96.5%, Clef-Flash 84.0%, CLIP 97.5%.
  • Extrusion: Jev 75.0%, Clef-Flash 71.0%, CLIP 71.5%.
  • Pipe staples: Jev 94.0%, Clef-Flash 99.0%, CLIP 57.0%.
  • Metallic pans: Jev 100%, Clef-Flash 88.5%, CLIP 64.1%.

An offline calculation reproduced all these values from the authors’ saved scores, fitting their probability-rescaling procedure on calibration rows only. It did not rerun model inference.

Check whether the baseline ranks badly or cuts badly

For pans, the authors report CLIP ranking images almost perfectly, with an area under the ranking curve of 0.999. Its fitted decision cutoff misses many defects. Using the baseline’s raw rule raises balanced accuracy to 91.0%, according to the study.

Platt scaling fits a logistic curve to adjust probabilities. On Jev’s pan results, the authors report calibration error rising from 0.114 to 0.205 after that adjustment. Correct class choices and reliable confidence are separate properties.

Match the defect rate to the factory

The test sets contain equal numbers of defective and clean images. Actual production may have far fewer defects.

The authors’ projection for Jev on extrusion gives only 5.7% precision at a 1% defect rate, assuming the measured class-specific error rates hold. Most alarms would then be false. That is a projection, not an observed factory deployment.

The revised experiment separates description tuning from calibration, but retains test sets the researchers had already seen in an earlier version. Some near-duplicate extrusion frames also remain a stated limitation.

Verification: inspected the README, analysis narrative, Jev image client, scoring guards and calibration/evaluation code; independently recomputed the headline accuracies from saved artifacts. No live API calls, image inspection, model execution or performance reproduction was performed.