[TCSVT2025] FMTrack: Frequency-aware Interaction and Multi-Expert Fusion for RGB-T Tracking

This is an official pytorch implementation of the 2025 IEEE Transactions on Circuits and Systems for Video Technology paper:

FMTrack: Frequency-aware Interaction and Multi-Expert Fusion for RGB-T Tracking
(accepted by IEEE Transactions on Circuits and Systems for Video Technology, DOI: 10.1109/TCSVT.2025.3601598)

The paper can be downloaded from IEEE Xplore and ResearchGate

The models and raw results can be downloaded from [BaiduYun] and [GitHub].

The tracking demos are displayed on the Bilibili.

Proposed modules

• FIN in Line 270 of vit.py
• MEFM in Line 128 of odtrack.py

Requirements

pip install -r environment.yml

Results

RGB-T Tracking

RGB-T Datasets (SR/PR)	TBSI (TPAMI25)	FMTrack256 (ours)
RGBT210	0.625 / 0.853	0.636 / 0.883
RGBT234	0.637 / 0.871	0.661 / 0.898
LasHeR	0.556 / 0.692	0.576 / 0.727
VTUAV-ST	0.672 / 0.810	0.728 / 0.857

Set project paths

Run the following command to set paths for this project

python tracking/create_default_local_file.py --workspace_dir . --data_dir ./data --save_dir ./output

After running this command, you can also modify paths by editing these two files

lib/train/admin/local.py  # paths about training
lib/test/evaluation/local.py  # paths about testing

Data Preparation

Training datasets download

• LasHeR: [Link]
• VTUAV-ST: : [Link]

Put the tracking datasets in ./data. It should look like this:

${PROJECT_ROOT}
 -- data
     -- LasHeR
         |-- test
         |-- train
         ...
     -- VTUAV-ST
         |-- test
         |-- train

Training

• Download pre-trained [MAE ViT-Base weights] and put it to $PROJECT_ROOT$/FMTrack/lib/models/pretrained_models.
• Download RGB Tracker pre-trained weights [ODTrack], put it to $PROJECT_ROOT$/FMTrack/lib/models/pretrained_models, and rename 'ODTrack_256_ep0300.pth.tar' to 'ODTrack_256_Pretrained.pth.tar'.

1.Training with one GPU.

cd /$PROJECT_ROOT$/FMTrack
CUDA_VISIBLE_DEVICES=0 python tracking/train.py --script odtrack --config baseline_256_lasher --save_dir ./output --mode single --nproc_per_node 1

2.Training with multiple GPUs.

cd /$PROJECT_ROOT$/FMTrack
CUDA_VISIBLE_DEVICES=0,1 python tracking/train.py --script odtrack --config baseline_256_lasher --save_dir ./output --mode multiple --nproc_per_node 2

Before training, please make sure the data path in local.py is correct.

Evaluation

Download the model FMTrack, extraction code: x2w9. Add the model to $PROJECT_ROOT$/FMTrack/output/checkpoints/train/.

python tracking/test.py --tracker_name odtrack --tracker_param baseline_256_lasher --dataset lasher_test --runid 15 --threads 4 --num_gpus 2
python tracking/analysis_results.py --tracker_name odtrack --tracker_param baseline_256_lasher --dataset_name lasher_test --runid 15

• We recommend the official evaluation toolkit for RGBT210, RGBT234, LasHeR, VTUAV !!!.

Before evaluation, please make sure the data path in local.py is correct.

Test FLOPs, and Speed

python tracking/profile_model.py --script odtrack --config baseline_256_lasher

Acknowledgement

The code based on the ODTrack, FreqFusion, and DRSformer.

We would like to express our sincere thanks to the contributors.

Citation:

If you find this work useful for your research, please cite the following papers:

@ARTICLE{10220112,
  author={Yuanliang Xue,Guodong Jin,Bineng Zhong,Tao Shen,Lining Tan,Chaocan Xue,Yaozong Zheng},
  journal={IEEE Transactions on Circuits and Systems for Video Technology}, 
  title={FMTrack: Frequency-aware Interaction and Multi-Expert Fusion for RGB-T Tracking}, 
  year={2025},
  doi={10.1109/TCSVT.2025.3601598}}

If you have any questions about this work, please contact with me via xyl_507@outlook.com