DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning(ECCV-2022 Oral)

This repository contains the Official Pytorch Implementation for DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning

@article{gao2021disco,
  title={DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning},
  author={Yuting Gao, Jia-Xin Zhuang, Shaohui Lin, Hao Cheng, Xing Sun, Ke Li, Chunhua Shen},
  journal={European Conference on Computer Vision(ECCV)},
  year={2022}
}

If the project is useful to you, please give us a star. ⭐️

Framework

Checkpoints

Teacher Models

Architecture	Self-supervised Methods	Model Checkpoints
ResNet152	MoCo-V2	ResNet152-checkpoint_0799.pth.tar
ResNet101	MoCo-V2	ResNet101-checkpoint_0199.pth.tar
ResNet50	MoCo-V2	ResNet50-checkpoint_0199.pth.tar

For teacher models such as ResNet-50*2 etc, we use their official implementation, which can be downloaded from their github pages.

Student Models by DisCo

Teacher/Students	Efficient-B0	Efficient-B1	ResNet-18	ResNet-34	MobileNet-v3	Vit-Tiny	XCiT-Tiny
ResNet-50	ResNet50-EfficientB0-checkpoint_0199.pth.tar	DisCo-R50-Effb1.pth.tar	ResNet50-ResNet18-checkpoint_0199.pth.tar	DisCo-R50-R34.pth.tar	DisCo-R50-Mob.pth.tar	-	-
ResNet-101	ResNet101-EfficientB0-checkpoint_0199.pth.tar	DisCo-R101-Effb1.pth.tar	ResNet101-ResNet18-checkpoint_0199.pth.tar	DisCo-R101-R34.pth.tar	DisCo-R101-Mob.pth.tar	-	-
ResNet-152	ResNet152-EfficientB0-checkpoint_0199.pth.tar	DisCo-R152-Effb1.pth.tar	ResNet152-ResNet18-checkpoint_0199.pth.tar	DisCo-R152-R34.pth.tar	DisCo-R152-Mob.pth.tar	-	-
ResNet-50*2	ResNet50w2-EfficientB0-checkpoint_0199.pth.tar	DisCo-RN50x2-Effb1.pth.tar	ResNet50w2-ResNet18-checkpoint_0199.pth.tar	DisCo-RN50x2-R34.pth.tar	DisCo-RN50x2-Mob.pth.tar	-	-
ViT-Small	-	-	-	-	-	DisCo-ViT-small-ViT-tiny.pth	-
XCiT-Small	-	-	-	-	-	-	XCiT-Small-XCiT-Tiny-checkpoint.pth

Requirements

• Python3

• Pytorch 1.6+

• Detectron2

• 8 GPUs are preferred

• ImageNet, Cifar10/100, VOC, COCO

Reproduction

Commands can be found on Reproduction.

Thanks

Code heavily depends on MoCo-V2, Detectron2.