Dynamic Slimmable Network (DS-Net)

November 30, 2021 · View on GitHub

This repository contains PyTorch code of our paper: Dynamic Slimmable Network (CVPR 2021 Oral).

Architecture of DS-Net. The width of each supernet stage is adjusted adaptively by the slimming ratio ρ predicted by the gate.

Accuracy vs. complexity on ImageNet.

Pretrained Supernet

Supernet Checkpoint

Here is a summary of sub-networks performance of the pretrained supernet:

Subnetwork	0	1	2	3	4	5	6	7	8	9	10	11	12	13
MAdds	133M	153M	175M	200M	226M	255M	286M	319M	355M	393M	433M	475M	519M	565M
Top-1 (%)	70.1	70.4	70.8	71.2	71.6	72.0	72.4	72.7	73.0	73.3	73.6	73.9	74.1	74.6
Top-5 (%)	89.4	89.6	89.9	90.2	90.3	90.6	90.9	91.0	91.2	91.4	91.5	91.7	91.8	92.0

Usage

1. Requirements

Install PyTorch 1.2.0+, for example:

conda install -c pytorch pytorch torchvision

Install pytorch-image-models 0.3.2, for example:
```
pip install timm==0.3.2
```
Download ImageNet from http://image-net.org/. Move validation images to labeled subfolders using following script: https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.shvalprep.sh

2. Stage I: Supernet Training

For example, train dynamic slimmable MobileNet supernet with 8 GPUs (takes about 2 days):

python -m torch.distributed.launch --nproc_per_node=8 train.py /PATH/TO/ImageNet -c ./configs/mobilenetv1_bn_uniform.yml

3. Stage II: Gate Training

Modify resume: in configs/mobilenetv1_bn_uniform_reset_bn.yml to your supernet checkpoint. Recalibrate BN before gate training

python -m torch.distributed.launch --nproc_per_node=8 train.py /PATH/TO/ImageNet -c ./configs/mobilenetv1_bn_uniform_reset_bn.yml

Modify resume: in configs/mobilenetv1_bn_uniform_gate.yml to your supernet checkpoint after BN recalibration or our pretrained Supernet Checkpoint. Start gate training
```
python -m torch.distributed.launch --nproc_per_node=8 train.py /PATH/TO/ImageNet -c ./configs/mobilenetv1_bn_uniform_gate.yml
```

Citation

If you use our code for your paper, please cite:

@inproceedings{li2021dynamic,
  author = {Changlin Li and
            Guangrun Wang and
            Bing Wang and
            Xiaodan Liang and
            Zhihui Li and
            Xiaojun Chang},
  title = {Dynamic Slimmable Network},
  booktitle = {CVPR},
  year = {2021}
}