CyberFreezeDev
Loading
A big model teaching a small one a narrow job, so it runs cheaper and faster.
A smaller student model learns to copy a larger teacher, but only on the task you care about, so it can be many times smaller.
Model distillation, or knowledge distillation, is a training method where a smaller student model learns to reproduce the behaviour of a larger teacher model. The idea was popularised in the 2015 paper Distilling the Knowledge in a Neural Network by Hinton, Vinyals and Dean.
The student does not need to match the teacher on everything. It only needs to match it on the task you care about, which is why a distilled model can be many times smaller and still do that task well.
The student learns from the teacher's full probabilities, not just its top answer.
In the classic method the student learns from the teacher's full probability distribution over answers, not just the top answer. Those probabilities carry information such as "this is probably a cat, slightly possibly a fox". A raised temperature softens the distribution so more of that information shows.
The teacher writes answers for many prompts, and the student is fine-tuned on them.
With language models it is common to have the teacher write answers, explanations or reasoning for many prompts, then fine-tune the student on those outputs.
DeepSeek's R1-Distill models and DistilBERT are well-known distilled models.
DeepSeek released R1-Distill models: smaller Qwen and Llama based models fine-tuned on reasoning traces from the much larger DeepSeek-R1. DistilBERT is an older, widely used distilled version of BERT.
A distilled small model handles high-volume narrow tasks for much less.
Classifying millions of messages or documents with a large hosted model is expensive. A distilled small model can do the same narrow task for much less.
Small models answer faster, which matters in chat, voice and real-time checks.
Small models answer faster, which matters in chat, voice and real-time checks.
It fits on one GPU or an edge device, keeping data in house.
A distilled model can fit on a single GPU or an edge device, keeping data in house.
A large model labels past support emails, staff correct a sample, and a small open model trained on them handles the daily volume.
A company routes support emails into twelve categories. It runs a large model over a few thousand past emails to label them, has staff correct a sample, and fine-tunes a small open-weight model on the result. The small model runs on its own server, handles the daily volume, and sends low-confidence emails to the large model or a person.
Some providers forbid training competing models on their outputs. Check first.
Some providers' terms forbid using their outputs to train competing models. Check before you distil from a hosted model.
A student built for one task is not a general assistant.
A student distilled for one task is not a general assistant. Test it outside its task and you will see the gap.
The student learns errors too, so have people correct a sample of labels.
The student learns errors too. Have people correct a sample of the teacher's labels.
Distillation trains a new, smaller model. Quantization keeps the same model and stores its weights with fewer bits. They are often combined: distil to a small model, then quantize it to fit the hardware.
For a narrow task, a few thousand good examples often go a long way. Quality and variety matter more than volume. Keep a separate real test set to check the result.
On the task it was trained for it can come close. On everything else it is usually weaker. Measure it on your own examples before switching.
Last updated