neuron-selectivity-transfer

This code is a re-implementation of the imagenet classification experiments in the paper Like What You Like: Knowledge Distill via Neuron Selectivity Transfer .

Citation

If you use our code in your research or wish to refer to the baseline results, please use the following BibTeX entry.

@article{NST2017
  author =   {Zehao Huang and Naiyan Wang},
  title =    {Like What You Like: Knowledge Distill via Neuron Selectivity Transfer},
  journal =  {arXiv preprint arXiv:1707.01219},
  year =     {2017}
}

Implementation

This code is implemented by a modified MXNet which supports ResNeXt-like augmentation. (This version of MXNet does not support cudnn7)

ImageNet data preparation

Download the ImageNet dataset and create pass through rec (following tornadomeet's repository but using unchange mode)

Run

• bash init.sh
• modify config/cfgs.py
• python train.py

Results on ImageNet-1k

Single Model, Single Crop Validation Error:

Method	Model	Top-1	Top-5	Download
Student	Inception-BN	25.74	8.07	Dropbox
NST (Poly kernel)	Inception-BN	24.81	7.55	Dropbox
NST (Poly kernel) + KD	Inception-BN	24.34	7.11	Dropbox
NST (Poly kernel) + KD	A modified ResNet-50	21.05	5.56	Dropbox

Note: The symbol of our modified ResNet-50 (following SENet Appendix B) is available in Dropbox. The mean RGB for our modified ResNet-50 is [123.68, 116.28, 103.53], std is [58.395, 57.12, 57.375]. For Inception-BN, we don't need to do data pre-processing since we add a BN layer in the beginning of the network.