GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval

This repository is the official PyTorch implementation of our AAAI 2024 paper GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval.

Catalogue

• 1. Getting Started
• 2. Run
• 3. Trained Models
• 4. Results
• 5. Citation

Getting Started

1. Clone this repository:

git clone https://github.com/haungmozhi9527/GMMFormer.git
cd GMMFormer

2. Create a conda environment and install the dependencies:

conda create -n prvr python=3.9
conda activate prvr
conda install pytorch==1.9.0 cudatoolkit=11.3 -c pytorch -c conda-forge
pip install -r requirements.txt

3. Download Datasets: All features of TVR, ActivityNet Captions and Charades-STA are kindly provided by the authors of MS-SL.

4. Set root and data_root in config files (e.g., ./Configs/tvr.py).

Run

To train GMMFormer on TVR:

cd src
python main.py -d tvr --gpu 0

To train GMMFormer on ActivityNet Captions:

cd src
python main.py -d act --gpu 0

To train GMMFormer on Charades-STA:

cd src
python main.py -d cha --gpu 0

Trained Models

We provide trained GMMFormer checkpoints. You can download them from Baiduyun disk.

Dataset	ckpt
TVR	Baidu disk
ActivityNet Captions	Baidu disk
Charades-STA	Baidu disk

Results

Quantitative Results

For this repository, the expected performance is:

Dataset	R@1	R@5	R@10	R@100	SumR
TVR	13.9	33.3	44.5	84.9	176.6
ActivityNet Captions	8.3	24.9	36.7	76.1	146.0
Charades-STA	2.1	7.8	12.5	50.6	72.9

Citation

If you find this repository useful, please consider citing our work:

@inproceedings{wang2023gmmformer,
  title={GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval},
  author={Wang, Yuting and Wang, Jinpeng and Chen, Bin and Zeng, Ziyun and Xia, Shu-Tao},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2024}
}