AutoProg

Modified from VOLO and Token Labeling, not the final version.

This project contains code of our paper Automated Progressive Learning for Efficient Training of Vision Transformers (CVPR 2022).

AutoProg achieves efficient training by automatically increasing the training overload on-the-fly.

AutoProg can accelerate ViTs training by up to 85.1% with no performance drop.

[New!] Check out the extension of this work on diffusion models! AutoProg-Zero [ arXiv | code ].

1. Requirements

• torch>=1.7.0 (1.8.0 is tested and recommended); torchvision>=0.8.0; timm==0.4.5; tlt==0.1.0; pyyaml; apex-amp

• Docker is recommended as this repo requires some outdated packages. All the requirements will be automatically installed via Dockerfile. Install Docker, then run:

Docker build Dockerfile

• data prepare: ImageNet with the following folder structure, you can extract imagenet by this script.

│imagenet/
├──train/
│  ├── n01440764
│  │   ├── n01440764_10026.JPEG
│  │   ├── n01440764_10027.JPEG
│  │   ├── ......
│  ├── ......
├──val/
│  ├── n01440764
│  │   ├── ILSVRC2012_val_00000293.JPEG
│  │   ├── ILSVRC2012_val_00002138.JPEG
│  │   ├── ......
│  ├── ......

2. Train

• As we use token labeling, please download the token labeling data in Google Drive or BaiDu Yun (password: y6j2), details about token labling are in here.

• Train volo_d1 with AutoProg for 100 epochs: first modify the data paths in scripts/train_auroprog.sh, then run:

sh scripts/train_autoprog.sh

Citation

If you use our code for your paper, please cite:

@inproceedings{li2022autoprog,
  author = {Li, Changlin and 
            Zhuang, Bohan and 
            Wang, Guangrun and
            Liang, Xiaodan and
            Chang, Xiaojun and
            Yang, Yi},
  title = {Automated Progressive Learning for Efficient Training of Vision Transformers},
  booktitle = {CVPR},
  year = 2022,
}

LICENSE

This repo is under the Apache-2.0 license.