IBM Technology’s Aadam Mulla gives an introductory walkthrough of knowledge distillation: training a smaller student model to imitate signals from a larger teacher model.
What the model learns
- A large language model assigns probabilities to possible next tokens. A hard training label identifies one observed next token; soft labels preserve how the teacher weights alternatives.
- In Mulla’s illustrative “cat sat on the …” example, the teacher favors “mat” but still assigns some weight to “floor” or “chair.” The percentages on screen are invented examples, not measured output.
- The student can train toward the teacher’s probability distribution rather than a single winning answer. Relative probabilities of alternatives are often called dark knowledge.
What gets smaller, and what might be lost
- Fewer model parameters generally mean less storage and computation, but actual device fit and speed depend on memory, architecture, precision and hardware.
- A smaller student usually loses some capability; distillation tries to preserve performance on chosen tasks at lower inference cost. It does not transfer a teacher’s knowledge without a training dataset, compute or evaluation.
- The video illustrates soft-label distillation, one method among several. Check the specific student on real tasks and the intended device before assuming it matches the teacher.
“Soft labels say what’s right, what’s close and what’s completely off target.” — Aadam Mulla