Model distillation is a training technique that transfers the knowledge from a large and expensive model (the teacher) to a smaller and more efficient one (the student). Instead of training the small model solely with the original labels, it learns to mimic the teacher's outputs, including its probability distributions (so-called soft labels), which contain richer information about the relationship between classes.
This technique matters because it allows models to be deployed in resource-constrained environments —mobile, browsers, or edge devices— without sacrificing too much quality. Its main advantages are:
- Lower inference cost and energy consumption.
- Faster response speed.
- Lighter models that are easier to distribute.
A practical example is DistilBERT, a reduced version of BERT that maintains around 97% of its performance with approximately 40% fewer parameters. It is worth noting that distillation rarely preserves 100% of the quality: there is a trade-off between size and precision that must be adjusted according to the use case.