Jack Lu and colleagues trained language models by learning which existing weights to set to zero. Their Deletion Tuning leaves the other weights untouched. The saved change is a compressed keep-or-delete mask layered over the original model.
- On Llama 8B math questions, the authors report 83.55% accuracy versus 83.24% for full fine-tuning, which updates every weight. On six held-out math tests with Qwen 7B, the corresponding averages are 52.32% versus 51.89%.
- The Llama 8B mask takes 8.4 MB versus a 30,633 MB full checkpoint, under the paper’s storage assumptions. Fewer than 0.05% of all model weights are zeroed in its main reinforcement-learning runs.
- On six earlier tasks, the deletion-trained Llama models stayed within one accuracy point of their starting performance. Full tuning ended 1.3–3.1 points below it. A one-shot pruning control did not reproduce the learning gains.
The results also extend to the largest models tested under supervised training and self-distillation. Smaller models sometimes lag full tuning, and individual math tests vary even where the average matches. The paper measures compressed storage, not faster inference; training still needs a trainable value for every targeted gate weight. Its retention test adds one task, so learning a sequence of tasks without forgetting remains untested.