SelM [[Paper]](https://doi.org/10.1145/3664647.3680926)

Overview

Environment Prepare

Our Code was tested upon a conda environment.

You can install conda by this link Conda and then create an environment as follows:

conda create -n selm python=3.9 

conda activate selm

We use Pytorch 2.0.1 with CUDA-11.7 as our default setting, install Pytorch by pip as below

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2

Notice : Mamba-ssm Link require CUDA 11.6+ , you might have to update your CUDA.

for other required packages:

pip install -r requirements.txt

Dataset and Pretrained Backbone

For AVSBench Dataset ,please refer to this link AVSBench to download the datasets

For Pretrained Backbone(ResNet50、PVT-v2、VGGish),please refer to this link to download.

You can place the dataset and pretrained backbone to the directory data pretrained backbone Notice : Don't forget to change the paths of data and model in config.py

Pretrained Model

You can download our pretrained SelM models by Google Drive and place it to the directory pretrained model

Train

For S4 and MS3 settings, we supply single-gpu train, run the command below :

#S4
cd avs_s4
bash train.sh

#MS3
cd avs_ms3
bash train.sh

Note that for AVSS setting, we supply muti-gpu train, to train SelM on 8 GPUs run:

cd avss
bash train.sh

Test

For test, remember to change the path of weights ,run:

#S4
cd avs_s4
bash test.sh

#MS3
cd avs_ms3
bash test.sh

#AVSS
cd avss
bash test.sh

Acknowledgement

This repo is based on AVSBench,RIS-DMMI,CGFormer,many thanks to these wonderful works.

Citation

If you are interested in our work, you can cite our work by below bibtex, thank you !

@inproceedings{li2024selm,
  title={SelM: Selective Mechanism based Audio-Visual Segmentation},
  author={Li, Jiaxu and Yu, Songsong and Wang, Yifan and Wang, Lijun and Lu, Huchuan},
  booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
  pages={3926--3935},
  year={2024}
}