Implementations
May 24, 2026 ยท View on GitHub
MXNet
PyTorch
- Attention Transfer
- Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
- Interpreting Deep Classifier by Visual Distillation of Dark Knowledge
- Mean teachers are better role models
- Relational Knowledge Distillation
- Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons
- Fast Human Pose Estimation Pytorch
- MEAL: Multi-Model Ensemble via Adversarial Learning
- MEAL-V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks
- Using Teacher Assistants to Improve Knowledge Distillation
- A Comprehensive Overhaul of Feature Distillation
- Contrastive Representation Distillation
- Transformer model distillation
- TinyBERT
- Data Efficient Model Compression
- Channel Distillation
- Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
- MGD: Matching Guided Distillation
- torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation
- Knowledge Distillation on SSD
- distiller: A large scale study of Knowledge Distillation
- Knowledge-Distillation-Zoo: Pytorch implementation of various Knowledge Distillation (KD) methods
- A PyTorch implementation for exploring deep and shallow knowledge distillation (KD) experiments with flexibility
- Neural Network Distiller by Intel AI Lab: a Python package for neural network compression research.
- KD_Lib: A Pytorch Knowledge Distillation library for benchmarking and extending works in the domains of Knowledge Distillation, Pruning, and Quantization.
- Vision Transformer Distillation
- Cross-Layer Distillation with Semantic Calibration
- Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation
- Distilling Knowledge via Knowledge Review
- Hierarchical Self-supervised Augmented Knowledge Distillation
- Causal Distillation for Language Models
- UniversalNER
- MobileSAM
- Logit-Standardization-KD
- Delayed Eps-Shrinking for Faster Once-For-All Training
- Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
- EchoDFKD
- Autoregressive Distillation of Diffusion Transformers (ARD)
- Simple Unsupervised Knowledge Distillation With Space Similarity
Lua
Torch
- Distilling knowledge to specialist ConvNets for clustered classification
- Sequence-Level Knowledge Distillation, Neural Machine Translation on Android
- cifar.torch distillation
- ENet-SAD
Theano
- FitNets: Hints for Thin Deep Nets
- Transfer knowledge from a large DNN or an ensemble of DNNs into a small DNN
Lasagne + Theano
Tensorflow
- Deep Model Compression: Distilling Knowledge from Noisy Teachers
- Distillation
- An example application of neural network distillation to MNIST
- Data-free Knowledge Distillation for Deep Neural Networks
- Inspired by net2net, network distillation
- Deep Reinforcement Learning, knowledge transfer
- Knowledge Distillation using Tensorflow
- Knowledge Distillation Methods with Tensorflow
- Zero-Shot Knowledge Distillation in Deep Networks in ICML2019
- Knowledge_distillation_benchmark via Tensorflow2.0
Caffe
- Face Model Compression by Distilling Knowledge from Neurons
- KnowledgeDistillation Layer (Caffe implementation)
- Knowledge distillation, realized in caffe
- Cross Modal Distillation for Supervision Transfer
- Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection
- Knowledge Distillation via Instance Relationship Graph
Keras
- Knowledge distillation with Keras
- keras google-vision's distillation
- Distilling the knowledge in a Neural Network