CMAD: Correlation-Aware and Modalities-Aware Distillation for Multimodal Sentiment Analysis with Missing Modalities

The code for CMAD: Correlation-Aware and Modalities-Aware Distillation for Multimodal Sentiment Analysis with Missing Modalities, which is accepted in ICCV 25.

The Framework of CMAD:

Figure: Overview of the proposed CMAD framework. It consists of a student model, a teacher model, and two key modules: Correlation-Aware Feature Distillation (CAFD) and Modalities-Aware Regularization (MAR). CAFD ensures feature matching between student-teacher pairs and correlation alignment across samples between student-teacher and teacher-teacher representations, while MAR dynamically adjusts the weight of each modality combinations based on its difficulty.

Datasets:

Please move the following datasets into directory ./datasets/.

The CMU-MOSEI dataset can be downloaded according to MIB and MAG through the following link:

pip install gdown
gdown https://drive.google.com/uc?id=1VJhSc2TGrPU8zJSVTYwn5kfuG47VaNQ3

Rename the pkl as mosei.pkl.

The IEMOCAP dataset can be downloaded according to MuLT.

Please put the files into directory ./datasets/.

Prerequisites:

* Python 3.8.10
* CUDA 11.5
* pytorch 1.12.1+cu113
* sentence-transformers 3.1.1
* transformers 4.30.2

Note that the torch version can be changed to your cuda version, but please keep the transformers==4.30.2 as some functions will change in later versions.

Pretrained model:

Downlaod the BERT-base , and put into directory ./BERT-EN/.

Run CMAD

You can train the teacher model from scratch using scripts in /CMAD/CMAD_sentiment/Teacher_Model, you can also try to download the checkpoint (Extraction code: px6s) we provided and put in directory /CMAD/CMAD_sentiment/Teacher_Model.

For MOSEI dataset, please run the following code in /CMAD/CMAD_sentiment/Student_Model:

python3 stu_config_mosei.py --dataset='mosei' --begin_epoch=20 --d_l=96 --delta=0.1 --depth=5 --gamma=1 --latent_dim=96 --learning_rate=2e-5 --n_epochs=80 --tau=0.2 --te_layers=2 --temperature=7.0 --train_batch_size=128

Citation:

Please cite our paper if you find our work useful for your research:

@inproceedings{zhuang2025cmad,
  title={CMAD: Correlation-Aware and Modalities-Aware Distillation for Multimodal Sentiment Analysis with Missing Modalities},
  author={Zhuang, Yan and Liu, Minhao and Bai, Wei and Zhang, Yanru and Zhang, Xiaoyue and Deng, Jiawen and Ren, Fuji},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={4626--4636},
  year={2025}
}

Acknowledgement

Thanks to MIB , MAG, DiCMoR, IMDer, GCNet, LNLN, HKT, LFMIM and MMANet for their great help to our codes and research.