Awesome-self-supervised-multimodal-learning

August 16, 2024 · View on GitHub

PRs WelcomeAwesome

A curated list of awesome self-supervised multimodal learning resources. Check our survey paper for details!

@article{zong2024self,
  title={Self-Supervised Multimodal Learning: A Survey},
  author={Zong, Yongshuo and Mac Aodha, Oisin and Hospedales, Timothy},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2024},
  publisher={IEEE}
}

Table of Contents

Overview

Taxonomy: Multimodal learning, which aims to understand and analyze information from multiple modalities, has achieved substantial progress in the supervised regime in recent years. However, the heavy dependence on data paired with expensive human annotations impedes scaling up models. Meanwhile, given the availability of large-scale unannotated data in the wild, self-supervised learning has become an attractive strategy to alleviate the annotation bottleneck. Building on these two directions, self-supervised multimodal learning (SSML) provides ways to learn from raw multimodal data. In this survey, we provide a comprehensive review of the state-of-the-art in SSML, in which we elucidate three major challenges intrinsic to self-supervised learning with multimodal data: (1) learning representations from multimodal data without labels, (2) fusion of different modalities, and (3) learning with unaligned data. We then detail existing solutions to these challenges. Specifically, we consider (1) objectives for learning from multimodal unlabeled data via self-supervision, (2) model architectures from the perspective of different multimodal fusion strategies, and (3) pair-free learning strategies for coarse-grained and fine-grained alignment.

Learning Paradigms: An example illustrating the self-supervised vision and language pretraining prior to downstream supervised learning for visual question answering is shown below. (a) supervised multimodal learning, and (b) self-supervised multimodal learning: Top, self-supervised pretraining without manual annotations; Bottom, supervised fine-tuning or linear readout for downstream tasks.

  • Multimodal machine learning: A survey and taxonomy.

  • Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions.

  • Deep multimodal learning: A survey on recent advances and trends.

    • IEEE signal processing magazine 2017 [paper]

  • Multimodal research in vision and language: A review of current and emerging trends.

  • Self-Supervised Representation Learning: Introduction, advances, and challenges.

    • IEEE Signal Processing Magazine 2022 [paper]

  • Self-supervised learning: Generative or contrastive.

  • Self-supervised visual feature learning with deep neural networks: A survey.

  • Vision-language pre-training: Basics, recent advances, and future trends.

Objectives

Instance Discrimination

In the context of multimodal learning, instance discrimination often aims to determine whether samples from two input modalities are from the same instance, i.e., paired. By doing so, it attempts to align the representation space of the paired modalities while pushing the representation space of different instance pairs further apart. There are two types of instance discrimination objectives: contrastive and matching prediction, depending on how the input is sampled.

  • Learning transferable visual models from natural language supervision.

  • Self-supervised multimodal versatile networks.

  • End-to-end learning of visual representations from uncurated instructional videos.

  • Scaling up visual and vision-language representation learning with noisy text supervision.

  • Contrastive Multiview Coding.

  • Audioclip: Extending Clip to Image, Text and Audio.

  • VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding.

  • CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval.

  • PointCLIP: Point Cloud Understanding by CLIP.

  • Image-and-Language Understanding from Pixels Only.

  • Scaling Language-Image Pre-training via Masking.

  • COOKIE: Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation.

  • Slip: Self-supervision meets language-image pre-training.

  • Crossclr: Cross-modal contrastive learning for multi-modal video representations.

  • CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding.

  • Learnable PINs: Cross-Modal Embeddings for Person Identity.

  • Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.

  • Learning Video Representations using Contrastive Bidirectional Transformer.

  • Learning representations from audio-visual spatial alignment.

  • Sound Localization by Self-Supervised Time Delay Estimation.

  • Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations.

  • Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings.

  • Fine-grained Multi-Modal Self-Supervised Learning.

  • Self-supervised Feature Learning by Cross-modality and Cross-view Correspondences.

  • Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization.

  • Audio-Visual Instance Discrimination with Cross-Modal Agreement.

  • Look, Listen and Learn.

  • Objects that Sound.

  • Audio-Visual Scene Analysis with Self-Supervised Multisensory Features.

  • The Sound of Pixels.

  • The Sound of Motions.

  • Music Gesture for Visual Sound Separation.

  • Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning.

Clustering

Clustering methods assume that applying end-to-end trained clustering will lead to the grouping of the data by semantically salient characteristics. In practice, these methods iteratively predict the cluster assignments of the encoded representation, and use these predictions, also known as pseudo labels, as supervision signals to update the feature representation. Multimodal clustering provides the opportunity to learn multimodal representations and also improve conventional clustering by using each modality’s pseudolabels to supervise the other.

  • Self-Supervised Learning by Cross-Modal Audio-Video Clustering.

  • Labelling unlabelled videos from scratch with multi-modal self-supervision.

  • Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction.

  • u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality.

  • Deep Multimodal Clustering for Unsupervised Audiovisual Learning.

  • Self-labelling via simultaneous clustering and representation learning.

Masked Prediction

The masked prediction task can be either performed in an auto-encoding (similar to BERT) or an auto-regressive approach (similar to GPT).

  • VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning.

  • CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations.

  • Jointly Learning Visual and Auditory Speech Representations from Raw Data.

  • Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks.

  • SimVLM: Simple Visual Language Model Pretraining with Weak Supervision.

  • VideoBERT: A Joint Model for Video and Language Representation Learning.

  • Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks.

  • VL-BEiT: Generative Vision-Language Pretraining.

  • OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation.

  • SelfDoc: Self-Supervised Document Representation Learning.

  • Deep Bidirectional Language-Knowledge Graph Pretraining.

  • ERNIE: Enhanced Language Representation with Informative Entities.

  • VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding.

  • Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions.

Hybrid

  • Align before Fuse: Vision and Language Representation Learning with Momentum Distillation.

  • DM2C: Deep Mixed-Modal Clustering.

  • Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks.

  • Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training.

  • UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation.

  • ActBERT: Learning Global-Local Video-Text Representations.

  • MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound.

  • BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.

  • ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision.

  • UNITER: UNiversal Image-TExt Representation Learning.

  • VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts.

  • FLAVA: A Foundational Language And Vision Alignment Model.

  • Vlmixer: Unpaired vision-language pre-training via cross-modal cutmix.

  • ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks.

  • Unsupervised Vision-and-Language Pretraining via Retrieval-based Multi-Granular Alignment.

  • Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning.

  • Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs.

  • Multimodal Deep Autoencoder for Human Pose Recovery.

  • Self-supervised object detection from audio-visual correspondence.

  • Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos.

  • Self-Supervised Learning of Audio-Visual Objects from Video.

  • Coot: Cooperative hierarchical transformer for video-text representation learning.

  • Unpaired Image Captioning via Scene Graph Alignments.

Applications

State Representation Learning

  • State representation learning for control: An overview

  • Unsupervised Representation Learning in Deep Reinforcement Learning: A Review

  • Action-Conditional Video Prediction using Deep Networks in Atari Games

  • Recurrent World Models Facilitate Policy Evolution

  • Learning latent dynamics for planning from pixels

  • Learning to Poke by Poking: Experiential Learning of Intuitive Physics

  • Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

Healthcare

  • Multimodal biomedical AI.

  • MedCLIP: Contrastive Learning from Unpaired Medical Images and Text.

  • ContIG: Self-supervised multimodal contrastive learning for medical imaging with genetics.

  • CoMIR: Contrastive multimodal image representation for registration.

  • Contrastive learning of medical visual representations from paired images and text.

  • GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition.

  • Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning.

  • Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports.

Remote Sensing

  • Multimodal remote sensing benchmark datasets for land cover classification with a shared and specific feature learning model.

    • ISPRS Journal of Photogrammetry and Remote Sensing 2021 [paper] [code]

  • Self-Supervised SAR-Optical Data Fusion of Sentinel-1/-2 Images.

    • IEEE Transactions on Geoscience and Remote Sensing 2022 [paper]

  • Semi-Supervised Learning for Joint SAR and Multispectral Land Cover Classification.

    • IEEE Geoscience and Remote Sensing Letters 2021 [paper]

  • Self-Supervised Change Detection in Multiview Remote Sensing Images.

    • IEEE Transactions on Geoscience and Remote Sensing 2021 [paper] [code]

  • Self-Supervised Multisensor Change Detection.

    • IEEE Transactions on Geoscience and Remote Sensing 2021 [paper] [code]

  • Self-supervised Audiovisual Representation Learning for Remote Sensing Data.

Machine Translation

  • A Survey of Multilingual Neural Machine Translation.

    • ACM Computing Surveys 2019 [paper]

  • Unsupervised Machine Translation Using Monolingual Corpora Only.

  • Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation.

  • Visual Grounding in Video for Unsupervised Word Translation.

  • Multilingual Unsupervised NMT using Shared Encoder and Language-Specific Decoders.

  • The Missing Ingredient in Zero-Shot Neural Machine Translation.

Auto-driving

  • Multi-modal Sensor Fusion for Auto Driving Perception: A Survey.

  • Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data.

  • Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR.

  • There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge.

  • Unsupervised Learning of Depth, Optical Flow and Pose with Occlusion from 3D Geometry.

Robotics

  • Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks.

  • Self-Supervised Visual Terrain Classification From Unsupervised Acoustic Feature Learning.

    • IEEE Transactions on Robotics 2019 [paper]

  • Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks.

  • Two stream networks for self-supervised ego-motion estimation.

  • Connecting Touch and Vision via Cross-Modal Prediction.

Challenges

Resources

  • Contrastive Vision-Language Pre-training with Limited Resources.

  • Beyond neural scaling laws: beating power law scaling via data pruning.

  • Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm.

Robustness/Fairness

  • When and why Vision-Language Models behave like Bags-of-Words, and what to do about it?.

  • Robustness Analysis of Video-Language Models Against Visual and Language Perturbations.

  • Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty.

  • Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning.

    • 2022 IEEE Symposium on Security and Privacy (SP) 2022 [paper] [code]

  • Are Multimodal Models Robust to Image and Text Perturbations?.

  • A Study of Gender Impact in Self-supervised Models for Speech-to-Text Systems.

  • Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications.

  • Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning.

  • Multimodal datasets: misogyny, pornography, and malignant stereotypes.

  • On the opportunities and risks of foundation models.

  • Extracting Training Data from Diffusion Models.

  • Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models.

    • Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP) 2022 [paper]

  • Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers.

  • What makes for good views for contrastive learning?.

Summary of Common Multimodal Datasets

Image-Text Datasets

Name# Images# TextDomainTaskAccessGithub
COCO>330000>1.5MNatural imagesimage captioning, image-text retrievalLinkGithub
Flickr30k31,0005 sentences for each imageNatural imagesimage captioning, image-text retrievalLink-
FlickrStyle10K10,00010,000Natural imagesimage captioning (stylized), image-text retrievalLinkGithub
Flickr8k8,0005 for eachNatural imagesimage captioning, image-text retrievalLinkGithub
Flickr8k-CN8,0008,000Natural imagesimage captioning, image-text retrievalLinkGithub
SentiCap1671/15004892/3977Natural imagesimage captioning (stylized), image-text retrievalLink-
SBU Captions1M1MNatural imagesimage captioning, image-text retrievalLinkLink
Conceptual Captions3M3MNatural imagesimage captioning, image-text retrievalLinkGithub
AIC-ICC210K210KNatural imagesimage captioning, image-text retrievalLinkGithub
Wikipedia28662866documentimage captioning, image-text retrieval?Github
NUS-WIDE-10K10K10KNatural imagesimage captioning, image-text retrievalLink-
Yelp200,1006,990,280product reviewsummarizationLink-
VQA v2.0204,7211105904/11,059,040 (Q/A)Natural imagesVQALink-
ImageCLEF 2019 VQA-Med38253825MedicineVQALinkGithub
VCR110k290k/290k/290k (Q/A/Rationale)naturalvisual commonsense reasoning (VCR)LinkGithub
GD-VCR328886/886(Q/A)Geo-Diversevisual commonsense reasoning (VCR)LinkGithub
SNLI-VEDetailsNatural imagesVisual EntailmentLinkGithub
NLVR2107,292107,292Natural imagesnatural language for visual reasoningLinkGithub
NLVR9224492244synthetic imagesnatural language for visual reasoningLinkGithub
rendered SST2~1k~1kimage of textoptical character recognition (OCR)Link-
OCR-CC1.4M1.4MNatural imagesoptical character recognition (OCR)LinkGithub
Hateful Memes10k+10k+memesoptical character recognition (OCR)LinkGithub
CORD1K1kdocumentOCRLinkGithub
RefCOCO+19,992141,564Natural imagesVisual GroundingLinkGithub

Image-Text-Audio Datasets

Name# Images# TextDomainTaskAccessGithub
Localized Narratives848,749873,107naturalImage captioning, Paragraph generation, VQA, Phrase grounding etc.LinkGithub
open image0.6M0.6MnaturalImage captioning, detection, segmentation, VQA, etcLinkGithub

Video-Text Datasets

Name# Video / # clips# TextDomainTasklinkGithub
ActivityNet Captions20k: 100k100knaturalVideo Captioning, video-text retrievalLinkGithub
V2C9k27knatural (human action)Video Captioning, video-text retrievalLinkGithub
VATEX41.3k826knaturalVideo Captioning, video-text retrievalLinkGithub
YouCook22k:15.4k15.4kCookingVideo Captioning, video-text retrievalLinkGithub
Charades10k:10k27.8kIndoor activityVideo Captioning, video-text retrieval, action recognitionLink-
MSR-VTT7k:10k200knaturalVideo Captioning, video-text retrievalLinkGithub
MSVD2k:2k70knaturalVideo Captioning, video-text retrieval--
HowTo100M1.2M: 136M136Minstructional videoVideo Captioning, video-text retrieval, action locolizationLink-
TGIF102k: 102k126kanimated GIFsVideo Captioning, video-text retrievalLinkGithub
TACoS-MLevel185:25k75kCookingVideo Captioning, video-text retrievalLink-
CrossTask4.7K:-4.7kinstructionalTemporal action localizationLinkGithub
MiningYoutube20k:200k200kCookingTemporal action localizationLinkGithub
COIN11,827:-46,35412 different domainsAction SegmentationLinkGithub
Breakfast-:1126711267cookingAction SegmentationLink-
LSMDC200:128k128kMovieVideo Captioning, video-text retrievalLink-
HOMAGE1.75KIndoor activityActivity ClassificationLinkGithub

Video-Audio Datasets

Name# Video-audio# utteranceDomainTaskAccessGithub
SoundNet2MnaturalAudio-visual correspondenceLinkGithub
MUSIC714music instrumentsAudio-visual correspondenceLinkGithub
AVSpeech290kPersonAudio-visual correspondenceLinkGithub
URMP44music instrumentsAudio-visual correspondenceLink-
AV-Benchv1 ~5k, v2 ~7knaturalAudio-Visual Correspondence (AVC), Audio-Visual Event Localization (AVEL) and video parsing (AVVP), Sound Source Localization (SSL), etc.LinkGithub
AVE4143naturaltemporal localizationLinkGithub
360° video1146cameraSpatial Audio generationLinkGithub
openposepersonAudio-visual correspondence, music-to-video generationLinkGithub
LRS2-144481personspeech recognition, lips readingLinkGithub
LRS39506151819personspeech recognition, lips readingLink-

Point Cloud Datasets

Name# meshDomainTaskAccessGithub
ModelNet4012,311CAD modelsClassification, reconstructionLinkGithub
ShapeNet220,0003D modelsClassification, reconstructionLink-
ScanObjectNN2902real-world point cloudClassification, reconstructionLinkGithub

Image-Ridar Datasets

Name# images# points (M)DomainTasklinkGithub
Eigen split KITTTI7481+75181799auto drivingdetectionLink-
nuScenesauto driving3D detection and trackingLinkGithub
SemanticKITTI23201+203514549auto drivingsegmentationLinkGithub

Contribute

PR welcome using the following markdown format:

- Paper Name. 
  - *Conference Year*. [[paper]](link) [[code]](link)