MobileViTv3 : Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features [arXiv]
October 6, 2022 ยท View on GitHub
This repository contains MobileViTv3's source code for training and evaluation. It uses the CVNets library and is inspired by MobileViT (paper, code).
Installation and Training Models:
We recommend to use Python 3.8+ and PyTorch (version >= v1.8.0) with conda environment.
For setting-up the python environment with conda, see here.
MobileViTv3-S,XS,XXS
Download MobileViTv1 and replace the files provided in MobileViTv3-v1.
Conda environment used for training: environment_cvnet.yml.
Then install according to instructions provided in the downloaded repository.
For training, use training-and-evaluation readme provided in the downloaded repository.
MobileViTv3-1.0,0.75,0.5
Download MobileViTv2 and replace the files provided in MobileViTv3-v2.
Conda environment used for training: environment_mbvt2.yml
Then install according to instructions provided in the downloaded repository.
For training, use training-and-evaluation readme provided in the downloaded repository.
Trained models:
Download the trained MobileViTv3 models from here.
checkpoint_ema_best.pt files inside the model folder is used to generated the accuracy of models.
Low-latency models are build by reducing the number of MobileViTv3-blocks in 'layer4' from 4 to 2.
Please refer to the paper for more details.
Note that for the segmentation and detection, only the backbone architecture parameters are listed.
Classification
ImageNet-1K:
| Model name | Accuracy (%) | Parameters (Million) | FLOPs (Million) | Foldername |
|---|---|---|---|---|
| MobileViTv3-S | 79.3 | 5.8 | 1841 | mobilevitv3_S_e300_7930 |
| MobileViTv3-XS | 76.7 | 2.5 | 927 | mobilevitv3_XS_e300_7671 |
| MobileViTv3-XXS | 70.98 | 1.2 | 289 | mobilevitv3_XXS_e300_7098 |
| MobileViTv3-1.0 | 78.64 | 5.1 | 1876 | mobilevitv3_1_0_0 |
| MobileViTv3-0.75 | 76.55 | 3.0 | 1064 | mobilevitv3_0_7_5 |
| MobileViTv3-0.5 | 72.33 | 1.4 | 481 | mobilevitv3_0_5_0 |
ImageNet-1K using low-latency models:
| Model name | Accuracy (%) | Parameters (Million) | FLOPs (Million) | Foldername |
|---|---|---|---|---|
| MobileViTv3-S-L2 | 79.06 | 5.2 | 1651 | mobilevitv3_S_L2_e300_7906 |
| MobileViTv3-XS-L2 | 76.10 | 2.3 | 853 | mobilevitv3_XS_L2_e300_7610 |
| MobileViTv3-XXS-L2 | 70.23 | 1.1 | 256 | mobilevitv3_XXS_L2_e300_7023 |
Segmentation
PASCAL VOC 2012:
| Model name | mIoU | Parameters (Million) | Foldername |
|---|---|---|---|
| MobileViTv3-S | 79.59 | 7.2 | mobilevitv3_S_voc_e50_7959 |
| MobileViTv3-XS | 78.77 | 3.3 | mobilevitv3_XS_voc_e50_7877 |
| MobileViTv3-XXS | 74.04 | 2.0 | mobilevitv3_XXS_voc_e50_7404 |
| MobileViTv3-1.0 | 80.04 | 13.6 | mobilevitv3_voc_1_0_0 |
| MobileViTv3-0.5 | 76.48 | 6.3 | mobilevitv3_voc_0_5_0 |
ADE20K:
| Model name | mIoU | Parameters (Million) | Foldername |
|---|---|---|---|
| MobileViTv3-1.0 | 39.13 | 13.6 | mobilevitv3_ade20k_1_0_0 |
| MobileViTv3-0.75 | 36.43 | 9.7 | mobilevitv3_ade20k_0_7_5 |
| MobileViTv3-0.5 | 33.57 | 6.4 | mobilevitv3_ade20k_0_5_0 |
Detection MS-COCO:
| Model name | mAP | Parameters (Million) | Foldername |
|---|---|---|---|
| MobileViTv3-S | 27.3 | 5.5 | mobilevitv3_S_coco_e200_2730 |
| MobileViTv3-XS | 25.6 | 2.7 | mobilevitv3_XS_coco_e200_2560 |
| MobileViTv3-XXS | 19.3 | 1.5 | mobilevitv3_XXS_coco_e200_1930 |
| MobileViTv3-1.0 | 27.0 | 5.8 | mobilevitv3_coco_1_0_0 |
| MobileViTv3-0.75 | 25.0 | 3.7 | mobilevitv3_coco_0_7_5 |
| MobileViTv3-0.5 | 21.8 | 2.0 | mobilevitv3_coco_0_5_0 |
Citation
If you find this repository useful, please consider giving a star :star: and citation :mega::
@inproceedings{wadekar2022mobilevitv3,
title = {MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features},
author = {Wadekar, Shakti N. and Chaurasia, Abhishek},
doi = {10.48550/ARXIV.2209.15159},
year = {2022}
}