Algorithms

December 10, 2025 · View on GitHub

Attention Distillation
Adversarial Distillation
Multi-teacher Distillation
Cross-modal Distillation
Graph-based Distillation
Adaptive Distillation
Contrastive Distillation

Attention Distillation

Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer, ICLR 2017, :link: :octocat:
Knowledge Transfer with Jacobian Matching, ICML 2018, :link:
Class Attention Map Distillation for Efficient Semantic Segmentation, MVIP 2020, :link:
Double Similarity Distillation for Semantic Image Segmentation, TIP 2021, :link:
Efficient Semantic Segmentation via Self-Attention and Self-Distillation, T-ITS 2022, :link:
Class Attention Transfer for Semantic Segmentation, AICAS 2022, :link: :octocat:
Hierarchical Multi-Attention Transfer for Knowledge Distillation, ACM MM 2023, :link:
Class Attention Transfer Based Knowledge Distillation, CVPR 2023, :link: :octocat:
Attention and feature transfer based knowledge distillation, Nature 2023, :link:
Student-friendly knowledge distillation, Knowledge-Based Systems 2024, :link:
Channel-spatial knowledge distillation for efficient semantic segmentation, Pattern Recognition 2024, :link: :octocat:
SAKD: Sparse attention knowledge distillation, Image and Video Computing 2024, :link:
Knowledge Diffusion for Distillation, NeurIPS 2024, :link: :octocat:
LAKD-Activation Mapping Distillation Based on Local Learning, arXiv 2024, :link:
Attention-guided Feature Distillation for Semantic Segmentation, CVIU 2025, :link: :octocat:

Adversarial Distillation

Adversarial Data-free Distillation

Teacher-student compression with generative adversarial networks, arXiv 2018, :link: :octocat:
Data-Free Adversarial Distillation, arXiv 2019, :link: :octocat:
Lifelong GAN: Continual Learning for Conditional Image Generation, ICCV 2019, :link:
Data-free learning of student networks, ICCV 2019, :link: :octocat:
Zero-shot knowledge transfer via adversarial belief matching, NeurIPS 2019, :link: :octocat:
Data-free knowledge amalgamation via group-stack dual-gan, CVPR 2020, :link:
Data-Free Network Quantization With Adversarial Knowledge Distillation, CVPR Workshop 2020, :link:
Dual Discriminator Adversarial Distillation for Data-free Model Compression, IJMLC 2022, :link:
Momentum adversarial distillation: Handling large distribution shifts in data-free knowledge distillation, NeurIPS 2022, :link:
Adversarial Data Augmentation for Task-Specific Knowledge Distillation of Pre-trained Transformers, AAAI 2022, :link:
Learning To Retain While Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation, CVPR 2023, :link:
Out of thin air: Exploring data-free adversarial robustness distillation, AAAI 2024, :link:
Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization, AAAI 2024, :link:
DERD: Data-free Adversarial Robustness Distillation through Self-adversarial Teacher Group, ACM Multimedia 2024, :link:

Discriminator in the Loop

Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks, arXiv 2017, :link:
Adversarial learning of portable student networks, AAAI 2018, :link:
Adversarial distillation for efficient recommendation with external knowledge, ACM TOIS 2018, :link:
Training student networks for acceleration with conditional adversarial networks, BMVC 2018, :link:
Adversarial network compression, ECCV 2018, :link:
Kdgan: Knowledge distillation with generative adversarial networks, NeurIPS 2018, :link:
Exploiting the ground-truth: An adversarial imitation based knowledge distillation approach for event detection, AAAI 2019, :link:
Meal: Multi-model ensemble via adversarial learning, AAAI 2019, :link: :octocat:
Knowledge squeezed adversarial network compression, arXiv 2019, :link:
structured knowledge distillation for semantic segmentation, CVPR 2019, :link: :octocat:
Adversarial Distillation for Learning with Privileged Provisions, IEEE TPAMI 2019, :link:
Hierarchical Knowledge Squeezed Adversarial Network Compression, AAAI 2020, :link:
Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer, arXiv 2020, :link:
AMLN: Adversarial-based Mutual Learning Network for Online Knowledge Distillation, ECCV 2020, :link:
Intra-class Feature Variation Distillation for Semantic Segmentation, ECCV 2020, :link: :octocat:
Feature-map-level online adversarial knowledge distillation, ICML 2020, :link:
Gan-knowledge distillation for one-stage object detection, IEEE Access 2020, :link:
KTAN: Knowledge Transfer Adversarial Network, IJCNN 2020, :link:
Lightning fast video anomaly detection via multi-scale adversarial distillation, Elsevier CVIU 2024, :link: :octocat:

Generative Adversarial Network Distillation

Compressing gans using knowledge distillation, arXiv 2019, :link:
MineGAN: effective knowledge transfer from GANs to target domains with few images, CVPR 2020, :link: :octocat:
Gan compression: Efficient architectures for interactive conditional gans, CVPR 2020, :link: :octocat:
Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation, CVPR 2022, :link: :octocat:

Diffusion Distillation

Sdxl-lightning: Progressive adversarial diffusion distillation, arXiv 2024, :link: :octocat:
CAD: Photorealistic 3D Generation via Adversarial Distillation, CVPR 2024, :link: :octocat:
Adversarial Score Distillation: When score distillation meets GAN, CVPR 2024, :link: :octocat:
ACT-Diffusion: Efficient Adversarial Consistency Training for One-step Diffusion Models, CVPR 2024, :link: :octocat:
Learning Differentially Private Diffusion Models via Stochastic Adversarial Distillation, ECCV 2024, :link:
Adversarial Diffusion Distillation, ECCV 2024, :link:

Adversarial Robust Distillation

Adversarially Robust Distillation, AAAI 2020, :link: :octocat:
Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better, ICCV 2021, :link: :octocat:
AGKD-BML: Defense Against Adversarial Attack by Attention Guided Knowledge Distillation and Bi-Directional Metric Learning, ICCV 2021, :link: :octocat:
Enhanced Accuracy and Robustness via Multi-Teacher Adversarial Distillation, ECCV 2022, :link: :octocat:
Reliable Adversarial Distillation with Unreliable Teachers, ICLR 2022, :link:
Indirect Gradient Matching for Adversarial Robust Distillation, arXiv 2023, :link:
Boosting Accuracy and Robustness of Student Models via Adaptive Adversarial Distillation, CVPR 2023, :link: :octocat:
Adversarial Local Distribution Regularization for Knowledge Distillation, WACV 2023, :link:
Adversarial Distillation Based on Slack Matching and Attribution Region Alignment, CVPR 2024, :link: :octocat:
Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples, CVPR 2024, :link:
PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor, CVPR 2024, :link: :octocat:
Dynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledge, ECCV 2024, :link: :octocat:
Adversarially Robust Distillation by Reducing the Student-Teacher Variance Gap, ECCV 2024, :link:

Multi-teacher Distillation

Amalgamating knowledge towards comprehensive classification, AAAI 2019, :link:
Knowledge amalgamation from heterogeneous networks by common feature learning, IJCAI 2019, :link: :octocat:
Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation, ICCV 2019, :link: :octocat:
Unifying Heterogeneous Classifiers with Distillation, CVPR 2019, :link:
A Two-Teacher Framework for Knowledge Distillation, ISNN 2019, :link:
Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning, ECCV 2020, :link:
Ensemble Knowledge Distillation for Learning Improved and Efficient Networks, ECAI 2020, :link:
FEED: Feature-level Ensemble for Knowledge Distillation, ECAI 2020, :link:
Robust Semantic Segmentation With Multi-Teacher Knowledge Distillation, IEEE Access 2021, :link:
Knowledge distillation guided by multiple homogeneous teachers, Information Science 2022, :link:
MTED: multiple teachers ensemble distillation for compact semantic segmentation, Neural Computing 2023, :link:
Multi-teacher knowledge distillation based on joint Guidance of Probe and Adaptive Corrector, Neural Networks 2023, :link:
Knowledge Amalgamation for Object Detection With Transformers, TIP 2023, :link: :octocat:
Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation, ICML 2023, :link: :octocat:
Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning, ICME 2023, :link: :octocat:
ATMKD: adaptive temperature guided multi-teacher knowledge distillation, Multimedia Systems 2024, :link:
Amalgamating Knowledge for Comprehensive Classification with Uncertainty Suppression, ISCAS 2024, :link:
Relation-Based Multi-Teacher Knowledge Distillation, IJCNN 2024, :link:
DE-MKD: Decoupled Multi-Teacher Knowledge Distillation Based on Entropy, Mathematics 2024, :link:
Improving Knowledge Distillation via Cross-Modal Insights from CLIP, ICASSP 2025, :link: :octocat:
back arrowGo to ICLR 2025 Conference homepage Single Teacher, Multiple Perspectives: Teacher Knowledge Augmentation for Enhanced Knowledge Distillation, ICLR 2025, :link:
Enriching Knowledge Distillation with Cross-Modal Teacher Fusion, Arxiv 2025, :link:

Cross Modal Distillation for Supervision Transfer, CVPR 2016, :link: :octocat:
Modality distillation with multiple stream networks for action recognition, ECCV 2018, :link: :octocat:
Learning with Privileged Information via Adversarial Discriminative Modality Distillation, TPAMI 2019, :link: :octocat:
Cross-Modal Knowledge Distillation for Action Recognition, ICIP 2019, :link:
3D-to-2D Distillation for Indoor Scene Parsing, CVPR 2021, :link: :octocat:
EvDistill: Asynchronous Events To End-Task Learning via Bidirectional Reconstruction-Guided Cross-Modal Knowledge Distillation, CVPR 2021, :link: :octocat:
Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection, ECCV 2022, :link: :octocat:
Robust Cross-Modal Representation Learning with Progressive Self-Distillation, CVPR 2022, :link:
UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird's-Eye View, CVPR 2023, :link: :octocat:
DistillBEV: Boosting Multi-Camera 3D Object Detection with Cross-Modal Knowledge Distillation, ICCV 2023, :link: :octocat:
X3KD: Knowledge Distillation Across Modalities, Tasks and Stages for Multi-Camera 3D Object Detection, CVPR 2023, :link:
Efficient RGB-T Tracking via Cross-Modality Distillation, CVPR 2023, :link:
Decomposed Cross-Modal Distillation for RGB-Based Temporal Action Detection, CVPR 2023, :link:
STXD: structural and temporal cross-modal distillation for multi-view 3D object detection, NeurIPS 2023, :link:
C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation, CVPR 2024, :link:
CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillation, CVPR 2024, :link: :octocat:
Xkd: Cross-modal knowledge distillation with domain alignment for video representation learning, AAAI 2024, :link: :octocat:
Radocc: Learning cross-modality occupancy knowledge through rendering assisted distillation, AAAI 2024, :link:
CM-MaskSD: Cross-Modality Masked Self-Distillation for Referring Image Segmentation, TMM 2024, :link:

Graph-based Distillation

Graph Distillation for Action Detection with Privileged Modalities, ECCV 2018, :link: :octocat:
Better and Faster: Knowledge Transfer from Multiple Self-supervised Learning Tasks via Graph Distillation for Video Classification, IJCAI 2018, :link:
Knowledge Transfer Graph for Deep Collaborative Learning, ACCV 2020, :link: :octocat:
Learning Student Networks via Feature Embedding, IEEE TNNLS 2020, :link:

Relational Knowledge Distillation via Graph

A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning, CVPR 2017, :link:
Relational Knowledge Distillation, CVPR 2019, :link: :octocat:
Similarity-Preserving Knowledge Distillation, ICCV 2019, :link:
Correlation Congruence for Knowledge Distillation, ICCV 2019, :link:
Knowledge Distillation via Instance Relationship Graph, CVPR 2019, :link: :octocat:
Graph-based Knowledge Distillation by Multi-head Attention Network, BMVC 2019, :link:
Binarized Collaborative Filtering with Distilling Graph Convolutional Networks, IJCAI 2019, :link:
Spatio-Temporal Graph for Video Captioning With Knowledge Distillation, CVPR 2020, :link: :octocat:
Heterogeneous Knowledge Distillation Using Information Flow Modeling, CVPR 2020, :link: :octocat:
Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning, IEEE TNNLS 2020, :link: :octocat:
Distilling Knowledge from Well-Informed Soft Labels for Neural Relation Extraction, AAAI 2020, :link:
Distilling Holistic Knowledge With Graph Neural Networks, ICCV 2021, :link: :octocat:
Deep Structured Instance Graph for Distilling Object Detectors, ICCV 2021, :link: :octocat:
Dark Reciprocal-Rank: Teacher-to-student Knowledge Transfer from Self-localization Model to Graph-convolutional Neural Network, IEEE ICRA 2021, :link: :octocat:
GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent Inference, MICCAI 2021, :link: :octocat:
Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural Network, AAAI 2021, :link: :octocat:

Graph Neural Network Distillation

Distilling Knowledge From Graph Convolutional Networks, CVPR 2020, :link: :octocat:
Graph few-shot learning via knowledge transfer, AAAI 2020, :link: :octocat:
Tinygnn: Learning efficient graph neural networks, ACM SIGKDD 2020, :link:
Reliable data distillation on graph convolutional network, ACM SIGMOD 2020, :link:
Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework, ACM WWW 2021, :link: :octocat:
Graph-free knowledge distillation for graph neural networks, IJCAI 2021, :link: :octocat:
MulDE: Multi-teacher knowledge distillation for low-dimensional knowledge graph embeddings, ACM WWW 2021, :link:
On self-distilling graph neural network, IJCAI 2021, :link:
Edge: Enriching knowledge graph embeddings with external text, NAACL 2021, :link:
ROD: Reception-aware Online Distillation for Sparse Graphs, ACM SIGKDD 2021, :link: :octocat:
Graph-less neural networks: Teaching old mlps new tricks via distillation, ICLR 2022, :link: :octocat:
Cold brew: Distilling graph node representations with incomplete or missing neighborhoods, ICLR 2022, :link: :octocat:
On representation knowledge distillation for graph neural networks, IEEE TNNLS 2022, :link: :octocat:
Knowledge distillation improves graph structure augmentation for graph neural networks, NeurIPS 2022, :link: :octocat:
Lte4g: Long-tail experts for graph neural networks, ACM CIKM 2022, :link: :octocat:
Sail: Self-augmented graph contrastive learning, AAAI 2022, :link:
Compressing Deep Graph Neural Networks via Adversarial Knowledge Distillation, ACM SIGKDD 2022, :link: :octocat:
Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding, ACM WWW 2022, :link: :octocat:
FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks, ACM SIGKDD 2022, :link:
Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks, NeurIPS 2022, :link: :octocat:
Data-Free Adversarial Knowledge Distillation for Graph Neural Networks, IJCAI 2022, :link:
Multi-Scale Distillation from Multiple Graph Neural Networks, AAAI 2022, :link: :octocat:
Alignahead: Online Cross-Layer Knowledge Extraction on Graph Neural Networks, IJCNN 2022, :link: :octocat:
Iterative Graph Self-Distillation, ICLR 2023, :link:
T2-GNN: Graph Neural Networks for Graphs with Incomplete Features and Structure via Teacher-Student Distillation, AAAI 2023, :link:
Boosting Graph Neural Networks via Adaptive Knowledge Distillation, AAAI 2023, :link: :octocat:
RELIANT: Fair Knowledge Distillation for Graph Neural Networks, SDM 2023, :link: :octocat:
NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphs, ICLR 2023, :link: :octocat:
The Devil is in the Data: Learning Fair Graph Neural Networks via Partial Knowledge Distillation, ACM WSDM 2024, :link:
Online adversarial knowledge distillation for graph neural networks, Elsevier ESWA 2024, :link: :octocat:

Adaptive Distillation

On the Efficacy of Knowledge Distillation, ICCV 2019, :link:
Knowledge Distillation for Semantic Segmentation Using Channel and Spatial Correlations and Adaptive Cross Entropy, Sensors 2019, :link:
Channel-wise attention for knowledge distillation, arXiv 2020, :link:
Improved Knowledge Distillation via Teacher Assistant, NeurIPS 2020, :link: :octocat:
Heterogeneous Knowledge Distillation using Information Flow Modeling, ECCV 2020, :link: :octocat:
InDistill: Information flow-preserving knowledge distillation for model compression, arXiv 2022, :link:
Prune Your Model Before Distill It, ECCV 2022, :link: :octocat:
Channel-Correlation-Based Selective Knowledge Distillation, T-CDS 2022, :link: :octocat:
Adaptive Perspective Distillation for Semantic Segmentation, T-MAPI 2022, :link: :octocat:
Category correlation and adaptive knowledge distillation for compact cloud detection in remote sensing images, T-GRS 2022, :link:
Masked Distillation with Receptive Tokens, ICLR 2023, :link: :octocat:
RdimKD: Generic Distillation Paradigm by Dimensionality Reduction, arXiv 2023, :link:
Knowledge Distillation with Active Exploration and Self-Attention Based Inter-Class Variation Transfer for Image Segmentation, ICASSP 2023, :link:
Holistic Weighted Distillation for Semantic Segmentation, ICME 2023, :link: :octocat:
Enlightening the Student in Knowledge Distillation, ICASSP 2023, :link: :octocat:
Multi-teacher knowledge distillation based on joint Guidance of Probe and Adaptive Corrector, Neural Networks 2023, :link:
Attention and feature transfer based knowledge distillation, Nature 2023, :link:
Similarity Knowledge Distillation with Calibrated Mask, ICASSP 2024, :link:
Coordinate Attention Guided Dual-Teacher Adaptive Knowledge Distillation for image classification, ESWA 2024, :link: :octocat:
Prune Channel And Distill: Discriminative Knowledge Distillation For Semantic Segmentation, ICIP 2024, :link:
CS-KD: Confused Sample Knowledge Distillation for Semantic Segmentation of Aerial Imagery, ICIC 2024, :link:
Adaptive class token knowledge distillation for efficient vision transformer, Knowledge-Based Systems 2024, :link:
AFMPM: adaptive feature map pruning method based on feature distillation, IJMLC 2024, :link:
Coordinate Attention Guided Dual-Teacher Adaptive Knowledge Distillation for image classification, ESWA 2024, :link:
Maximizing discrimination capability of knowledge distillation with energy function, Knowledge-Based Systems 2024, :link: :octocat:
Adaptive Inter-Class Similarity Distillation for Semantic Segmentation, MTAP 2025, :link: :octocat:

Contrastive Distillation

Knowledge Distillation for Single Image Super-Resolution via Contrastive Learning, ACM 2024, :link:
ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model, arXiv 2024, :link:
Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation, NeurIPS 2023, :link: :octocat:
Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation, CVPR 2021, :link: :octocat:
Complementary Relation Contrastive Distillation, CVPR 2021, :link: :octocat:
Wasserstein Contrastive Representation Distillation, CVPR 2021, :link:
Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification, MICCAI 2021, :link: :octocat:
DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings, arXiv 2021, :link:
Contrastive Representation Distillation, arXiv 2019, :link: :octocat:
Dimensionality Reduction by Learning an Invariant Mapping, CVPR 2006, :link: