Knowledge Distillation
The process of transferring the knowledge of a large and complex model to a smaller model.
The small 'student' model in information distillation works to achieve similar performance with much less calculation source by mimicing the output potential distributions of the large 'student' model.