Experiments on SimMIM

Preparation

Download the teacher Swin-Small to <BASE_FOLDER>/checkpoints/swin_small_800e.pth.tar

Pre-Training

# 1024 batch-sizes on 8 GPUs:
cd projects/simmim
ssl_train -b 2048 -d 0-7 -e 400 -f ./simmim_exp.py --amp \
--exp-options exp_name=simmim/simmim_swin_tiny_400e

Pre-Training with Decoupled Distillation

# 1024 batch-sizes on 8 GPUs:
cd projects/simmim
ssl_train -b 2048 -d 0-7 -e 400 -f ./simmim_distill_exp.py --amp --exp-options \
teacher_ckpt_path="<BASE_FOLDER>/checkpoints/swin_small_800e.pth.tar" exp_name=simmim/simmim_swin_tiny_d2_distill_400e

Fine-Tuning

# 1024 batch-sizes on 8 GPUs:
cd projects/eval_tools
ssl_train -b 1024 -f finetuning_swin_exp.py --amp [--ckpt <checkpoint-path>] \
--exp-options pretrain_exp_name=simmim/simmim_swin_tiny_d2_distill_400e

Main Results

pre-train code	pre-train epochs	fine-tune recipe	fine-tune epoch	accuracy	ckpt
simmim	400	-	-	-	link
simmim	400	impr.	300	77.4	link
simmim_d2_distill	400	-	-	-	link
simmim_d2_distill	400	impr.	300	77.8	link