SKILL.PNG turns a task instruction into an optimized image attached to a model’s request. Maximilian Seeth and colleagues adjust the pixels through a frozen vision-language model, which accepts images and text. The model’s weights stay unchanged.
The image is a compact learned input, rather than a readable illustrated manual. Training requires access to the model’s internal gradients; using a finished image requires only an ordinary image input.
- Across four models and four task sets, image instructions significantly beat the SkillOpt text optimizer in nine of sixteen comparisons.
- Each image uses 64–256 image tokens per request. Text instructions average 937–1,963 tokens, but image optimization takes about 12.2–19.9 GPU hours per task set.
- Exact tool selection shows no significant gain from standalone images. Nemotron’s score on that task falls by 9.82 percentage points.
- A jointly optimized Qwen/Gemma image improves both participating models. Images do not improve models outside their training group.
Caveat: Main results disable reasoning, and task-format shortcuts may explain some gains. Turning reasoning back on makes benefits less consistent.
The public implementation supports inspecting training and evaluation. For practical use, shorter context must justify its training bill, weak portability and loss of human-readable instructions.