Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval
August 12, 2025 ยท View on GitHub
This repository contains the official PyTorch implementation of our AAAI 2025 paper, "Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval", based on GMMFormer v2.
The core implementation of our proposed framework is contained in the ./src/Models/frame_module/.
Contents
Getting Started
1. Clone this repository:
git clone https://github.com/junwoojinjin/SDM-PRVR.git
cd SDM-PRVR
2. Set the environment and install the dependencies:
conda create -n prvr python=3.9
conda activate prvr
# Install PyTorch (choose the correct CUDA version for your system)
conda install pytorch==1.9.0 cudatoolkit=11.3 -c pytorch -c conda-forge
pip install -r requirements.txt
3. Download Datasets
We use the pre-extracted features for TVR and ActivityNet Captions provided by the authors of MS-SL. Please download them from here.
4. Set root and data_root in config files (e.g., ./Configs/tvr.py).
Run
To train SDM on TVR:
cd src
python main.py -d tvr --gpu 0
To evaluate SDM on TVR:
python main.py -d tvr --gpu 0 --eval --resume ./checkpoints/sdm_gmmv2_tvr.pth
To train SDM on ActivityNet Captions:
cd src
python main.py -d act --gpu 0
To evaluate SDM on ActivityNet Captions:
python main.py -d act --gpu 0 --eval --resume ./checkpoints/sdm_gmmv2_act.pth
Trained Models
We provide trained SDM(ours) with GMMFormer_v2 checkpoints. You can download them from the Google Drive links.
| Dataset | ckpt |
|---|---|
| TVR | link |
| ActivityNet Captions | link |
Results
The expected performance of SDM on TVR and ActivityNet Captions is as follows:
| Dataset | R@1 | R@5 | R@10 | R@100 | SumR |
|---|---|---|---|---|---|
| TVR | 17.4 | 39.7 | 51.4 | 87.9 | 196.4 |
| ActivityNet Captions | 9.1 | 27.3 | 40.4 | 79.8 | 156.6 |
Reference
@inproceedings{jun2025bridging,
title={Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval},
author={Jun, WooJin and Moon, WonJun and Cho, Cheol-Ho and Jung, MinSeok and Heo, Jae-Pil},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={39},
number={4},
pages={4166--4174},
year={2025}
}
Acknowledgements
Our code is built upon the following repositories: