Model Distillation
Also: Knowledge Distillation, Teacher-Student Training
Model distillation trains a smaller student model to mimic a larger teacher model, transferring its behavior into a faster, cheaper, and more deployable form.
What It Is
Model distillation (also called knowledge distillation) is a technique for transferring the capabilities of a large, accurate model (the teacher) into a smaller, more efficient model (the student). Instead of training the student only on hard labels (the correct answer), distillation trains it to reproduce the teacher's richer outputs, such as full probability distributions, intermediate representations, or generated responses.
The key insight is that a teacher's soft outputs carry more information than a single label. When a teacher assigns 70% to "cat", 25% to "dog", and 5% to "fox", it reveals similarity relationships between classes that a hard label hides. The student learns these nuances, often reaching accuracy close to the teacher at a fraction of the size.
Why it matters
- Lower cost: Smaller models require less compute and memory, cutting inference bills.
- Faster inference: Reduced latency enables real time and on device use.
- Edge deployment: Distilled models can run on phones, browsers, or embedded hardware.
- Energy efficiency: Fewer parameters mean lower power consumption at scale.
How it is used in practice
1. Train or obtain a strong teacher model.
2. Run the teacher over a dataset to collect soft targets (logits, probabilities, or generated text).
3. Train the student to match these targets, often combined with the original labels.
4. A temperature parameter softens probability distributions to expose more detail.
5. Evaluate the student on accuracy, latency, and size, then iterate.
Variants include response based distillation (matching outputs), feature based distillation (matching hidden layers), and self distillation (a model teaching a copy of itself).
Concrete Example
A company runs a 70 billion parameter language model that answers support tickets accurately but costs too much per query. They use it as a teacher to generate thousands of high quality responses, then distill that behavior into a 7 billion parameter student. The student handles routine tickets at one tenth the cost with similar quality, while the large model is reserved for complex escalations.
Trade Offs
Distillation rarely matches the teacher perfectly, and the student inherits the teacher's biases and errors. Quality depends heavily on the distillation data and objective chosen.
See also
Frequently asked questions
What is model distillation in one sentence?
Model distillation is a training technique where a smaller model (the student) learns to reproduce the behaviour of a larger, more accurate model (the teacher). Instead of learning only from correct labels, the student is trained on the teacher's richer outputs, such as full probability distributions, hidden representations, or generated text. It is also called knowledge distillation or teacher-student training.
Why not just deploy the large model directly?
Because inference cost, latency, and hardware constraints often make it impractical. A distilled model uses less compute and memory, which cuts the bill per query, reduces response time enough for real-time use, and can run on phones, browsers, or embedded hardware. A common pattern is to serve routine traffic with the small model and reserve the large one for complex cases.
What is the difference between distillation and simply training a small model from scratch?
A small model trained from scratch only sees hard labels, one correct answer per example. Distillation gives it the teacher's soft outputs instead, which reveal similarity relationships between options: a teacher assigning 70% to "cat", 25% to "dog", and 5% to "fox" transmits information a single label hides. That extra signal is why a distilled student often lands close to the teacher's accuracy at a fraction of the size.
What are the steps of a distillation project?
Five steps: obtain or train a strong teacher, run it over a dataset to collect soft targets (logits, probabilities, or generated text), train the student to match those targets (often alongside the original labels), tune a temperature parameter that softens the distributions to expose more detail, then evaluate the student on accuracy, latency, and size and iterate.
What are the limits of a distilled model?
A student rarely matches its teacher exactly, and it inherits the teacher's biases and errors along with its capabilities. Final quality depends heavily on the distillation data and the objective chosen: response-based distillation matches outputs, feature-based distillation matches hidden layers, and self-distillation has a model teach a copy of itself. Plan an evaluation that measures the accuracy actually lost, not just the size gained.