MVBench

Download MVBench to ./eval/MVBench

torchrun --nproc_per_node=8 ./eval/eval_mvbench.py --model_path Hoar012/TDC-Qwen2-7B --model_name cambrian_qwen --version qwen --data_path eval/MVBench

PerceptionTest

Download the validation split of PerceptionTest and place it in ./eval/PerceptionTest

torchrun --nproc_per_node=8 ./eval/eval_perception.py --model_path Hoar012/TDC-Qwen2-7B --model_name cambrian_qwen --version qwen --data_path eval/PerceptionTest

EgoSchema

Download EgoSchema to ./eval/EgoSchema

torchrun --nproc_per_node=8 ./eval/eval_egoschema.py --model_path Hoar012/TDC-Qwen2-7B --model_name cambrian_qwen --version qwen --data_path eval/EgoSchema

Then submit the resulting .csv file to Kaggle.

MLVU

Download MLVU to ./eval/MLVU

torchrun --nproc_per_node=8 ./eval/eval_mlvu.py --model_path Hoar012/TDC-Qwen2-7B --model_name cambrian_qwen --version qwen --data_path eval/MLVU

Set the --use_lvcot flag to enable evaluation with LVCoT.

torchrun --nproc_per_node=8 ./eval/eval_mlvu.py --model_path Hoar012/TDC-Qwen2-7B --model_name cambrian_qwen --version qwen --data_path eval/MLVU --use_lvcot

VideoMME

Download VideoMME to ./eval/VideoMME

torchrun --nproc_per_node=8 ./eval/eval_videomme.py --model_path Hoar012/TDC-Qwen2-7B --model_name cambrian_qwen --version qwen --data_path eval/VideoMME --use_subtitle
torchrun --nproc_per_node=8 ./eval/eval_videomme.py --model_path Hoar012/TDC-Qwen2-7B --model_name cambrian_qwen --version qwen --data_path eval/VideoMME --use_subtitle --use_lvcot 

Evaluation on Audio-Visual Comprehension

Music-AVQA

Download the processed videos and audios from Music-AVQA to ./eval/Music-AVQA

torchrun --nproc_per_node=8 ./eval/eval_musicQA.py --model_path Hoar012/TDC-Qwen2-7B --model_name cambrian_qwen --version qwen --data_path eval/Music-AVQA --test_file data/AV_data/Music-AVQA/avqa-test.json

AVSD

Download the processed videos and audios from AVSD to ./eval/AVSD

torchrun --nproc_per_node=8 ./eval/eval_avsd.py --model_path Hoar012/TDC-Qwen2-7B --model_name cambrian_qwen --version qwen --data_path eval/AVSD --test_file eval/AVSD/avsd_val.json