🌟 Introduction

This repository is the official implementation of CMFS, a unified framework that leverages CLIP-guided modality interaction to mitigate noise in multi-modal image fusion and segmentation.

Infrared-visible image fusion and semantic segmentation are pivotal tasks for robust scene understanding under challenging conditions such as low light. However, existing methods often struggle with high noise, modality inconsistencies, and inefficient cross-modal interactions, limiting fusion quality and segmentation accuracy.

To address these challenges, we propose CMFS (CLIP-guided Multi-modal Fusion and Segmentation), a unified framework that leverages vision-language priors to enhance both fusion fidelity and semantic segmentation accuracy. By incorporating category text semantic guidance from CLIP, CMFS facilitates region-aware cross-modal interaction, improving the discrimination of foreground structures while suppressing modality-specific degradation.

🔬 Key Contributions

The proposed framework comprises:

• Region-aware Modal Interaction Alignment module that integrates a VMamba-based encoder, a shuffle layer, and CLIP-guided interaction to enhance foreground saliency and suppress noise.

• Frequency-Spatial Collaboration module that leverages selective scanning and integrates wavelet-, spatial-, and Fourier-domain features to achieve adaptive denoising.

• Low-rank Mixture-of-Experts with dynamic routing to improve region-specific fusion and pixel-level segmentation accuracy.

🔥 News

🔥 [June 24, 2025]: The paper and Code were released!
🔥 [April 29, 2025]: Our paper was accepted by IJCAI 2025!

🚀 QuickStart

Installation

Environment Preparation: cuda=11.8, python=3.11, torch=2.3.1, gcc=11.3(for SSM enviroment). Clone this repository, then create a cmfs conda environment and install the packages.

# clone repository
git clone https://github.com/SuGuilin/IJCAI2025-CMFS.git
# create conda env
conda create -n cmfs python=3.11
conda activate cmfs
# install packages
pip install -r requirements.txt
# install mamba
cd utils/dis_causal_conv1d && pip install -e . && cd ../..
cd utils/dis_mamba && pip install -e . && cd ../..

💡 Note: Installing Mamba may cost a lot of effort. If you encounter problems, we recommend following the official installation instructions provided by Mamba or issues in Mamba instead.

Dataset

After downloading the dataset(MFNet, FMB, PST900),

The FMB dataset requires additional processing. You need to write a script to write the training set and test set image names to train.txt and test.txt respectively, and then mix them together according to the folders Visible/, Infrared/, and Label/.

please create a /datasets folder in the project directory and place the downloaded data inside. The directory structure should be as follows:

IJCAI2025-CMFS
├── assets
├── configs
├── dataloader
├── datasets
│   ├── FMB
│   ├── MFNet
│   └── PST900
├── engine
├── experiment
├── models
├── pretrained
├── utils
├── eval.py
├── README.md
├── train.py
└── requirements.txt

💡 Note: If you wish to use a custom dataset path, simply modify the dataset_path field in configs/base_config.yaml accordingly and please orgnize the dataset folder in the following structure:

<datasets>
|-- <DatasetName1>
    |-- <RGBFolder>
        |-- <name1>.<ImageFormat>
        |-- <name2>.<ImageFormat>
        ...
    |-- <ModalXFolder>
        |-- <name1>.<ModalXFormat>
        |-- <name2>.<ModalXFormat>
        ...
    |-- <LabelFolder>
        |-- <name1>.<LabelFormat>
        |-- <name2>.<LabelFormat>
        ...
    |-- train.txt
    |-- test.txt
    |-- val.txt
|-- <DatasetName2>
|-- ...

train.txt/val.txt/test.txt contains the names of items in training/valing/testing set, e.g.:

<name1>
<name2>
...

Running

1. Please download the pretrained VMamba_Tiny_classification weights and put it under pretrained/.

2. Multi-GPU distributed training:

   NCCL_P2P_DISABLE=1 \
   CUDA_VISIBLE_DEVICES="0,1" \
   python -m torch.distributed.launch \
       --nproc_per_node=2 \
       --master_port=29588 \
       train.py \
       -d 0,1 \
       -p 29588 \
       -n "dataset_name"
   

   Here, dataset_name=mfnet/pst900/fmb, referring to the four datasets.

   • --nproc_per_node: specifies how many GPUs to use
   • -d: specifies the GPU device ID to use
   • -p: specifies the port for init_process_group
   • -n: specifies the dataset to use, supports the following options: mfnet, fmb, pdt900

3. Single-GPU training:

   CUDA_VISIBLE_DEVICES="0" torchrun -m --nproc_per_node=1 --master_port=29501 train.py -d 0 -n "dataset_name"
   

4. Results will be saved in experiment/dataset_name/exp/ folder，including checkpoint/、tb/ and train_output.log

Evaluation

1. Run the evaluation for segmentation by:

   CUDA_VISIBLE_DEVICES="0" python eval.py -n "dataset_name" -e "epoch_number"
   

2. Run the evaluation for fusion by:

   CUDA_VISIBLE_DEVICES="0" python eval_vision.py -n "dataset_name" -e "epoch_number"
   

✨ Acknowledgments

Our code is based on MRFS, VMamba and CLIP. We sincerely appreciate for their contributions.

🔗 Citation

Thank you for your interest in our work. If this work is useful to you, please cite it as follows:

@inproceedings{su2025cmfs,
  title={CMFS: CLIP-Guided Modality Interaction for Mitigating Noise in Multi-Modal Image Fusion and Segmentation},
  author={Guilin Su, Yuqing Huang, Chao Yang and Zhenyu He},
  booktitle={Proceedings of the Thirty-four International Joint Conference on Artificial Intelligence},
  year={2025}
}