Fully-connected Transformer for Multi-source Image Fusion

October 11, 2025 · View on GitHub

This repo is the official code of Fully-connected Transformer for Multi-source Image Fusion (TPAMI' 2024).

[Paper] [Project Page] [Data] [Model Zoo]

[UDL] [PanCollection] [HyperSpectralCollection] [MSIF]

News🔥🔥🔥

  • We will release the complete code within one month after the ICCV deadline.

  • :art: The FC-Former convers multiple multi-source image fusion scenes:

    • Multispectral and hyperspectral image fusion;
    • Remote sensing pansharpening;
    • Visible and infrared image fusion (VIS-IR);
    • Digital photographic image fusion: Multi-focus image fusion (MFF) and multi-exposure image fusion (MEF).

  • 🎁 We will release a new version of UDL, PanCollection. Furthermore, we also release repositories of HyperSpectralCollection, comming soon and MSIF, comming soon.

  • :eight_pointed_black_star: The FC-Former employs multilinear algebra to unify and generalize self-attention mechanisms. One of the generalized self-attention mechanisms is fully-connected self-attention (FCSA).

  • 👍 The FC-Former achieves the state-of-the-art performance with less parameter number and have a powerful potential in feature representation, which is a win-win situation.

Quick Start 🤗

  1. Install our basic training/inference repo with multiple Pytorch framework support. You can select one of backends: accelerate, lightning, transformers, and mmcv1.
pip install udl_vis --upgrade
  1. Install the [Task] repo by populating one of the following repos: pancollection, mhif msif.
pip install [Task] --upgrade
  1. Then in this repository:
pip install -e .

Train

Here, we take Huggingface accelerate as the backend, thus

sh  run_accelerate_ddp_pansharpening.sh

The script will call python_scripts/accelerate_pansharpening.py to run the FC-Former. More information you can see model.yaml

Inference

Inference the FC-Former to obtain final results, you only need to update the model.yaml as follows:

  eval : true # change false to true
  workflow:
    - ["test", 1]
    # - ["train", 10] # comment it

Test Your Metrics

Finally, we provide the correspoinding Matlab toolboxes to test the metrics on those tasks. Please check them in our repo.

Experiments

In this part, we conduct experiments in the following cases. All settings can be changed in dataset.yaml.

  • MHIF: CAVE and Harvard.
  • VIS-IR image fusion: TNO and RoadScene datasets.
  • Pansharpening: WorldView-3, GaoFen-2, QuickBird datasets.
  • Digital photographic image fusion: MFF-WHU, MEF-Lytro,MEF-SLICE, MEFB.

Citation

Please cite this project if you use datasets or the toolbox in your research.

@article{FCFormer,
  title={Fully-connected Transformer for Multi-source  Image Fusion},
  author={Xiao Wu, Zi-Han Cao, Ting-Zhu Huang, Liang-Jian Deng, Jocelyn Chanussot, and Gemine Vivone}
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2025},
  publisher={IEEE}
}
q
@inproceedings{Wu_2021_ICCV,
    author    = {Wu, Xiao and Huang, Ting-Zhu and Deng, Liang-Jian and Zhang, Tian-Jing},
    title     = {Dynamic Cross Feature Fusion for Remote Sensing Pansharpening},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {14687-14696}
}

@article{zhong2024ssdiff,
  title={SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening},
  author={Zhong, Yu and Wu, Xiao and Deng, Liang-Jian and Cao, Zihan},
  journal={arXiv preprint arXiv:2404.11537},
  year={2024}
}

@ARTICLE{duancvpr2024,
title={Content-Adaptive Non-Local Convolution for Remote Sensing Pansharpening},
author={Yule Duan, Xiao Wu, Haoyu Deng, Liang-Jian Deng*},
journal={IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR)},
year={2024}
}

@ARTICLE{dengijcai2023,
title={Bidirectional Dilation Transformer for Multispectral and Hyperspectral Image Fusion},
author={Shang-Qi Deng, Liang-Jian Deng*, Xiao Wu, Ran Ran, Rui Wen},
journal={International Joint Conference on Artificial Intelligence (IJCAI)},
year={2023}
}

@misc{PanCollection,
    author = {Xiao Wu, Liang-Jian Deng and Ran Ran},
    title = {"PanCollection" for Remote Sensing Pansharpening},
    url = {https://github.com/XiaoXiao-Woo/PanCollection/},
    year = {2022},
}

And some related works about image fusion may attract you:

@inproceedings{
    cao2024novel,
    title={A novel state space model with local enhancement and state sharing for image fusion},
    author={Cao, Zihan and Wu, Xiao and Deng, Liang-Jian and Zhong, Yu},
    booktitle={ACM Multimedia 2024 (ACM MM)},
    year={2024}
}

@article{cao2024neural,
  title={Neural Shr$\backslash$" odinger Bridge Matching for Pansharpening},
  author={Cao, Zihan and Wu, Xiao and Deng, Liang-Jian},
  journal={arXiv preprint arXiv:2404.11416},
  year={2024}
}

@article{cao2024diffusion,
  title={Diffusion model with disentangled modulations for sharpening multispectral and hyperspectral images},
  author={Cao, Zihan and Cao, Shiqi and Deng, Liang-Jian and Wu, Xiao and Hou, Junming and Vivone, Gemine},
  journal={Information Fusion},
  volume={104},
  pages={102158},
  year={2024},
  publisher={Elsevier}
}

License

This project is open sourced under GNU General Public License v3.0.