NKD and USKD

ICCV 2023 Paper: From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels

Train

#single GPU
python tools/train.py configs/distillers/imagenet/res18_sd_img.py

#multi GPU
bash tools/dist_train.sh configs/distillers/imagenet/res34_distill_res18_img.py 8

Transfer

# Tansfer the Distillation model into mmcls model
python pth_transfer.py --dis_path $dis_ckpt --output_path $new_mmcls_ckpt

Test

#single GPU
python tools/test.py configs/resnet/resnet18_8xb32_in1k.py $new_mmcls_ckpt --metrics accuracy

#multi GPU
bash tools/dist_test.sh configs/resnet/resnet18_8xb32_in1k.py $new_mmcls_ckpt 8 --metrics accuracy

Results

NKD

Model	Teacher	Baseline(Top-1 Acc)	+NKD(Top-1 Acc)	dis_config	weight
ResNet18	ResNet34	69.90	71.96 (+2.06)	config	baidu/one drive
MobileNet	ResNet50	69.21	72.58 (+3.37)	config	baidu/one drive
DeiT-Tiny	DeiT III-Small	74.42	76.68 (+2.26)	config
DeiT-Base	DeiT III-Large	81.76	84.96 (+3.20)	config

USKD

Model	Baseline(Top-1 Acc)	+tf-NKD(Top-1 Acc)	dis_config
MobileNet	69.21	70.38 (+1.17)	config
MobileNetV2	71.86	72.41 (+0.55)	config
ShuffleNetV2	69.55	70.30 (+0.75)	config
ResNet18	69.90	70.79 (+0.89)	config
ResNet50	76.55	77.07 (+0.52)	config
ResNet101	77.97	78.54 (+0.57)	config
RegNetX-1.6GF	76.84	77.30 (+0.46)	config
Swin-Tiny	81.18	81.49 (+0.31)	config
DeiT-Tiny	74.42	74.97 (+0.55)	config

Citation

@inproceedings{yang2023knowledge,
  title={From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels},
  author={Yang, Zhendong and Zeng, Ailing and Yuan, Chun and Li, Yu},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={17185--17194},
  year={2023}
}