Self-Supervised Learning

This repo contains a curated list of self-supervised learning papers with a focus on representation learning and clustering. Feel free to add pull requests or open issues to suggest papers.

Table of Contents

• Representation Learning
  • Analysis
  • Image-Level Representations
    • Contrastive Learning
    • Masked Image Modeling
    • Proxy tasks
    • Clustering
  • Dense Representations
• Image Clustering
• Geometry

Representation Learning

Analysis

• W. Van Gansbeke, S. Vandenhende, S. Georgoulis, L. Van Gool, Revisiting Contrastive Methods for Unsupervised Learning of Visual Representation, NeurIPS, 2021. [Code]
• R. Geirhos, K. Narayanappa, B. Mitzkus, M. Bethge, F. A. Wichmann, W. Brendel, On the surprising similarities between supervised and self-supervised models, ICLR, 2021.
• X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, J. Tang, Self-supervised Learning: Generative or Contrastive, Arxiv, 2020.
• L. Jing, Y. Tian, Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey, T-PAMI, 2020.
• S. Purushwalkam, A. Gupta, Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases, NIPS, 2020.
• Alejandro Newell, Jia Deng, How Useful is Self-Supervised Pretraining for Visual Tasks?, CVPR, 2020. [Code]
• Y. M. Asano, C. Rupprecht, A. Vedaldi, A critical analysis of self-supervision, or what we can learn from a single image, ICLR, 2020. [Code]

Image-Level Representation Learning

Contrastive Learning

• X. Chen, S. Xie, K. He, An Empirical Study of Training Self-Supervised Vision Transformers, ICCV, 2021.
• P. Goyal, M. Caron, B. Lefaudeux, M. Xu, P. Wang, V. Pai, M. Singh, V. Liptchinsky, I. Misra, A. Joulin, P. Bojanowski, Self-supervised Pretraining of Visual Features in the Wild, Arxiv, 2021.
• X. Chen, K. He, Exploring Simple Siamese Representation Learning, CVPR, 2021.
• J. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, M. Valko, Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning, NeurIPS, 2020. [Code]
• T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A Simple Framework for Contrastive Learning of Visual Representations, ICML, 2020. [Code]
• M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, M. Lucic, On Mutual Information Maximization for Representation Learning, ICLR, 2020. [Code]
• Y. Tian, D. Krishnan, P. Isola, Contrastive Multiview Coding, ECCV, 2020. [Code]
• K. He, H. Fan, Y. Wu, S. Xie, R. Girshick, Momentum Contrast for Unsupervised Visual Representation Learning, CVPR, 2020. [Code]
• I. Misra, L. Maaten, Self-Supervised Learning of Pretext-Invariant Representations, CVPR, 2020.
• O. Henaff, A. Razavi, C. Doersch, S. Eslami, A. Oord, Data-Efficient Image Recognition with Contrastive Predictive Coding, ICML, 2020.
• P. Bachman, R. D. Hjelm, W. Buchwalter, Learning Representations by Maximizing Mutual Information Across Views, NIPS, 2019. [Code]
• R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, Y. Bengio, Learning deep representations by mutual information estimation and maximization, ICLR, 2019. [Code]
• J. Huang, Q. Dong, S. Gong, X. Zhu, Unsupervised Deep Learning by Neighbourhood Discovery, ICML, 2019. [Code]
• A. Oord, Y. Li, O. Vinyals, Representation Learning with Contrastive Predictive Coding, Arxiv, 2018.
• Z. Wu, Y. Xiong and X. Y. Stella, D. Lin, Unsupervised Feature Learning via Non-parameteric Instance Discrimination, CVPR, 2018. [Code]
• Wang, Xiaolong and He, Kaiming and Gupta, Abhinav, Transitive Invariance for Self-supervised Visual Representation Learning, ICCV, 2017.
• Li, Dong and Hung, Wei-Chih and Huang, Jia-Bin and Wang, Shengjin and Ahuja, Narendra and Yang, Ming-Hsuan, Unsupervised Visual Representation Learning by Graph-based Consistent Constraints, ECCV, 2016. [Code]

Clustering

• M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, A. Joulin, Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, NIPS, 2020. [Code]
• Y. M. Asano, C. Rupprecht, A. Vedaldi, Self-labelling via simultaneous clustering and representation learning, ICLR, 2020. [Code]
• X. Yan, I. Misra, A. Gupta, D. Ghadiyaram, D. Mahajan, ClusterFit: Improving Generalization of Visual Representations, CVPR, 2020.
• M. Caron, P. Bojanowski, J. Mairal, A. Joulin, Unsupervised Pre-Training of Image Features on Non-Curated Data, ICCV, 2019. [Code]
• M. Caron, P. Bojanowski, A. Joulin, M. Douze, Deep Clustering for Unsupervised Learning of Visual Features, ECCV, 2018. [Code]
• J. Yang, D. Parikh, D. Batra, Joint Unsupervised Learning of Deep Representations and Image Clusters, CVPR, 2016. [Code]
• J. Xie, R. Girshick, A. Farhadi, Unsupervised Deep Embedding for Clustering Analysis, ICML, 2016. [Code]

Masked Image Modeling

• X. Dong, J. Bao, T. Zhang, D. Chen, W. Zhang, L. Yuan, D. Chen, F. Wen, N. Yu, PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers, Arxiv, 2022.
• R. Wang, D. Chen, Z. Wu, Y. Chen, X. Dai, M. Liu, Y. Jiang, L. Zhou, L. Yuan, BEVT: BERT Pretraining of Video Transformers, CVPR, 2022.
• H. Bao, L. Dong, F. Wei, BEIT: BERT Pre-Training of Image Transformers, Arxiv, 2021.
• K. He, X. Chen, S. Xie, Y. Li, P. Dollár, R. Girshick, Masked Autoencoders Are Scalable Vision Learners, Arxiv, 2021.
• Z. Xie Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, H. Hu SimMIM: A Simple Framework for Masked Image Modeling, Arxiv, 2021.
• M. Chen, A. Radford, R. Child, J. Wu, H. Jun, P. Dhariwal, D. Luan, I Sutskever, Generative Pretraining from Pixels, ICML, 2020.

Proxy Tasks

• X. Zhan, X. Pan, Z. Liu, D. Lin, C. C. Loy, Self-Supervised Learning via Conditional Motion Propagation, CVPR, 2019. [Code]
• Z. Feng, C. Xu, D. Tao, Self-Supervised Representation Learning by Rotation Feature Decoupling, CVPR, 2019. [Code]
• A. Kolesnikov, X. Zhai, L. Beye, Revisiting Self-Supervised Visual Representation Learning, CVPR, 2019. [Code]
• T. Chen, X. Zhai, M. Ritter, M. Lucic, N. Houlsby, Self-Supervised GANs via Auxiliary Rotation Loss, CVPR, 2019. [Code]
• L. Zhang, G. Qi, L. Wang, J. Luo, AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data, CVPR, 2019. [Code]
• X. Zhai, A. Oliver, A. Kolesnikov, L. Beyer, S4L: Self-Supervised Semi-Supervised Learning, ICCV, 2019. [Code]
• S. Gidaris, P. Singh, N. Komodakis, Unsupervised Representation Learning by Predicting Image Rotations, ICLR, 2018 [Code]
• S. Jenni, P. Favaro, Self-Supervised Feature Learning by Learning to Spot Artifacts, CVPR, 2018. [Code]
• M. Noroozi, A. Vinjimoor, P. Favaro, H. Pirsiavash, Boosting Self-Supervised Learning via Knowledge Transfer, CVPR, 2018.
• A. Mahendran, J. Thewlis, A. Vedaldi, Cross Pixel Optical-Flow Similarity for Self-Supervised Learning, ACCV, 2018.
• M. Noroozi, H. Pirsiavash, P. Favaro, Representation Learning by Learning to Count, ICCV, 2017. [Code]
• R. Zhang, P. Isola, A. Efros, Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction, CVPR, 2017. [Code]
• D. Pathak, R. Girshick, P. Dollar, T. Darrell, B. Hariharan, Learning Features by Watching Objects Move, CVPR, 2017. [Code]
• G. Larsson, M. Maire, G. Shakhnarovich, Colorization as a Proxy Task for Visual Understanding, CVPR, 2017. [Code]
• R. S. Cruz, B. Fernando, A. Cherian, S. Gould, DeepPermNet: Visual Permutation Learning, CVPR, 2017. [Code]
• H. Lee, J. Huang, M. K. Singh, M. Yang, Unsupervised Representation Learning by Sorting Sequences, ICCV, 2017. [Code]
• P. Bojanowski, A. Joulin, Unsupervised Learning by Predicting Noise, ICML, 2017. [Code]
• M. Noroozi, P. Favaro, Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles, ECCV, 2016. [Code]
• G. Larsson, M. Maire, G. Shakhnarovich, Learning Representations for Automatic Colorization, ECCV, 2016. [Code]
• R. Zhang, P. Isola, A. Efros, Colorful Image Colorization, ECCV, 2016. [Code]
• D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, A. Efros, Context Encoders: Feature Learning by Inpainting, CVPR, 2016. [Code]
• P. Agrawal, J. Carreira, J. Malik, Learning to See by Moving, ICCV, 2015. [Code]
• C. Doersch, A. Gupta, A. Efros, Unsupervised Visual Representation Learning by Context Prediction, ICCV, 2015. [Code]

Dense Representation Learning

• W. Van Gansbeke, S. Vandenhende, S. Georgoulis, L. Van Gool, Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals, Arxiv, 2021. [Code]
• X. Wang, R. Zhang, C. Shen, T. Kong, L. Li, Dense Contrastive Learning for Self-Supervised Visual Pre-Training, Arxiv, 2020.
• Z. Xie, Y. Lin, Z. Zhang, Y. Cao, S. Lin, H. Hu, Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning, Arxiv, 2020. [Code]
• A. Jabri, A, Owens, A. Efros, Space-Time Correspondence as a Contrastive Random Walk, NIPS, 2020. [Code]
• P. O. Pinheiro, A. Almahairi, R. Y. Benmalek, F. Golemo, A. Courville, Unsupervised Learning of Dense Visual Representations, NIPS, 2020.
• X. Ji, J. F. Henriques, A. Vedaldi, Invariant Information Clustering for Unsupervised Image Classification and Segmentation, ICCV, 2019. [Code]
• X. Wang, A. Jabri, A. Efros. Learning Correspondence from the Cycle-Consistency of Time, CVPR, 2019. [Code]
• T. Zhou, P. Krahenbuhl, M. Aubry, Q. Huang, A. Efros.Learning dense correspondence via 3d-guided cycle consistency, CVPR, 2016.

Image Clustering

• W. Van Gansbeke, S. Vandenhende, S. Georgoulis, M. Proesmans, L. Van Gool, SCAN: Learning to Classify Images without Labels, ECCV, 2020. [Code]
• X. Ji, J. F. Henriques, A. Vedaldi, Invariant Information Clustering for Unsupervised Image Classification and Segmentation, ICCV, 2019. [Code]
• J. Chang, L. Wang, G. Meng, S. Xiang, C. Pan, Deep Adaptive Image Clustering, ICCV, 2017. [Code]
• J. Yang, D. Parikh, D. Batra, Joint unsupervised learning of deep representations and image clusters, CVPR, 2016. [Code]
• J. Xie, R. Girshick, A. Farhadi, Unsupervised Deep Embedding for Clustering Analysis, ICML, 2016. [Code]

Geometry

• C. Godard, O. M. Aodha, M. Firman, G. J. Brostow, Digging into Self-Supervised Monocular Depth Prediction, ICCV, 2019. [Code]
• J. Wang, J. Jiao, L. Bao, S. He, Y. Liu, W. Liu, SelFlow: Self-Supervised Learning of Optical Flow, CVPR, 2019. [Code]
• C. Godard, O. <. Aodha, G. J. Brostow, Unsupervised Monocular Depth Estimation with Left-Right Consistency, CVPR, 2017. [Code]
• T. Zhou, M. Brown, N. Snavely, D. G. Lowe, Unsupervised Learning of Depth and Ego-Motion from Video, CVPR, 2017. [Code]