Randall Balestriero and Yann LeCun study when separate image and text models learn similar ways to represent a scene. They train text encoders using different captions of the same image as matching views, then compare them with models trained on altered copies of one caption or next-token prediction.
- On one image–text alignment test, their 109-million-parameter encoder scored .184 against .111 for a next-token model trained on the same texts. Qwen2.5-72B scored .154. These scores measure similarity between representations, not general assistant ability.
- In a separate comparison, the matching-view model reached a similar alignment score with 21.6 million training samples versus 900 million for next-token prediction. The runs used different settings, and the authors did not evaluate earlier next-token checkpoints.
- The paper proves a sufficient condition in an idealized setting. Real observations lose information, and the experiments do not satisfy all the theorem’s assumptions.
Caveat: Most large-scale matching captions came from an image captioner that had itself learned from image–text data. The comparison between training objectives uses the same texts, but the approach does not eliminate cross-modal information from the data pipeline.