Model Distillation
Aussi : Knowledge Distillation, Teacher-Student Training
Model distillation trains a smaller student model to mimic a larger teacher model, transferring its behavior into a faster, cheaper, and more deployable form.
What It Is
Model distillation (also called knowledge distillation) is a technique for transferring the capabilities of a large, accurate model (the teacher) into a smaller, more efficient model (the student). Instead of training the student only on hard labels (the correct answer), distillation trains it to reproduce the teacher's richer outputs, such as full probability distributions, intermediate representations, or generated responses.
The key insight is that a teacher's soft outputs carry more information than a single label. When a teacher assigns 70% to "cat", 25% to "dog", and 5% to "fox", it reveals similarity relationships between classes that a hard label hides. The student learns these nuances, often reaching accuracy close to the teacher at a fraction of the size.
Why it matters
- Lower cost: Smaller models require less compute and memory, cutting inference bills.
- Faster inference: Reduced latency enables real time and on device use.
- Edge deployment: Distilled models can run on phones, browsers, or embedded hardware.
- Energy efficiency: Fewer parameters mean lower power consumption at scale.
How it is used in practice
1. Train or obtain a strong teacher model.
2. Run the teacher over a dataset to collect soft targets (logits, probabilities, or generated text).
3. Train the student to match these targets, often combined with the original labels.
4. A temperature parameter softens probability distributions to expose more detail.
5. Evaluate the student on accuracy, latency, and size, then iterate.
Variants include response based distillation (matching outputs), feature based distillation (matching hidden layers), and self distillation (a model teaching a copy of itself).
Concrete Example
A company runs a 70 billion parameter language model that answers support tickets accurately but costs too much per query. They use it as a teacher to generate thousands of high quality responses, then distill that behavior into a 7 billion parameter student. The student handles routine tickets at one tenth the cost with similar quality, while the large model is reserved for complex escalations.
Trade Offs
Distillation rarely matches the teacher perfectly, and the student inherits the teacher's biases and errors. Quality depends heavily on the distillation data and objective chosen.