ISTASTrack: Bridging ANN and SNN via ISTA Adapter for RGB-Event Tracking

The official implementation for ISTASTrack: Bridging ANN and SNN via ISTA Adapter for RGB-Event Tracking.

Usage

Data Preparation

Download the training datasets FE240hz, VisEvent, COESOT and FELT. We modified FE108py to convert raw events in .aedat format into T event frames. Please modify ./prepare_data/stack_event.py to customize for different datasets.

Path Setting

Run the following command to set paths (see run.sh):

dataset=fe240
model=istastrack

python tracking/create_default_local_file.py --workspace_dir . --data_dir /data/MyData/ --save_dir ./output/${model}_${dataset} \

You can also modify paths by these two files:

./lib/train/admin/local.py  # paths for training
./lib/test/evaluation/local.py  # paths for testing

Training

Dowmload the pretrained OSTrack and Spikingformer checkpoints and put them under ./pretrained/. Run ./tracking/train.py

NCCL_P2P_LEVEL=NVL python tracking/train.py --script ${model} --config ${dataset} --save_dir ./output/${model}_${dataset} --mode multiple --nproc_per_node 2 \
2>&1 | tee -a train.log

Testing

Run ./tracking/test.py to generate predicted results and run ./tracking/analysis_results.py to evaluate results, as follows in run.sh:

CUDA_VISIBLE_DEVICES=0,1 python tracking/test.py ${model} ${dataset} --dataset ${dataset} --threads 8 --num_gpus 2 --debug 0

python tracking/analysis_results.py --model_name ${model} --dataset ${dataset}  --parameter_name ${dataset}

Note that our analysis_results exclude invalid frames where the target disappears, whereas we found that some other benchmarks discard these frames during evaluation but still include them when averaging results.

Acknowledgment

• This repo is modified based on BAT which is an exellent work, helps us to quickly implement our ideas.
• Thanks for the OSTrack, PyTracking, Spikingformer library.
• We thank VisualizerX for visualization support.

Citation