Enhancing Partially Relevant Video Retrieval with Robust Alignment Learning

This repository provides the PyTorch implementation of our EMNLP 2025 paper, "Enhancing Partially Relevant Video Retrieval with Robust Alignment Learning", based on GMMFormer v2.

TL;DR. We explicitly model the data uncertainty in PRVR to make retrieval more robust. We model the video and query embeddings as Gaussian distributions, where the variance measures the inherent uncertainty of each instance. Based on these distributional representations, we construct text and video proxies as multiple possible alignment candidates, allowing the model to capture diverse cross-modal relations. In addition, we weight word–frame similarities with learnable word confidence. RAL integrates seamlessly with PRVR baselines (e.g., MS‑SL, GMMFormer, GMMFormer v2) and achieves state‑of‑the‑art results on TVR and ActivityNet Captions.

The core implementation of our framework resides in ./src/Models/.

Contents

1. Method Overview
2. Getting Started
3. Run
4. Results
5. Reproducibility Notes
6. Citation
7. Acknowledgements

---

Method Overview

1) Multimodal Semantic Robust Alignment (MSRA)

• We collect a query support set containing all queries related to the video, obtaining its features $\text{Q}_s$ with rich contexts.
• Then, we apply multi-granularity aggregation to obtain holistic semantics, and generate distributional representations parameterized by mean vector $\boldsymbol{\mu}$ and variance vector $\boldsymbol{\sigma}$.
• A proxy matching loss $\mathcal{L}\_{PM}$ and a distribution alignment loss $\mathcal{L}_{\mathrm{DA}}$ are used to unify the video and text domains.

2) Confidence‑aware Set‑to‑Set Alignment (CSA)

• We adopt a confidence predictor to assign confidence weights to each word, which is used to adjust the word-frame similarity matrix for video retrieval.

Final objective.

$$\mathcal{L} = \lambda_1 \, \mathcal{L}_{\text{InfoNCE}} + \lambda_2 \, \mathcal{L}_{\text{Triplet}} + \lambda_3 \, \mathcal{L}_{\text{DA}} + \lambda_4 \, \mathcal{L}_{\text{PM}}.$$

---

---

Getting Started

1. Clone this repository

git clone https://github.com/zhanglong-ustc/RAL-PRVR.git
cd RAL-PRVR

2. Create environment & install dependencies

conda create -n prvr python=3.9
conda activate prvr
# Install PyTorch (select a CUDA version that matches your system)
conda install pytorch==1.9.0 cudatoolkit=11.3 -c pytorch -c conda-forge
pip install -r requirements.txt

3. Datasets We follow common practice and use pre-extracted visual/text features for TVR and ActivityNet Captions. You may reuse features released by prior work MS-SL. Please download them from here.
   Set root and data_root in the config files (e.g., ./Configs/tvr.py).

---

Run

RAL is plug‑and‑play and can be attached to PRVR backbones.

1. RAL Training

cd src
python main.py -d tvr --gpu 0

2. RAL Evaluate

After training, you can inference using the saved model on val or test set:

Below is an example of inference using RAL, which produces the best performance reported in the paper.

python main.py -d tvr --gpu 0 --eval --resume ./checkpoints/ral_gmmv2_tvr.pth

The model can also be downloaded from Here.

NOTE: The pretrained checkpoints are now available.

Results

Benchmarks: TVR and ActivityNet Captions.
Backbone: GMMFormer v2, GMMFormer and MS-SL.

---

Reproducibility Notes

• Features: ResNet + I3D for vision; RoBERTa for text (same as prior PRVR works).
• Optimizer: Adam, LR 1e-4, batch 128, train 100 epochs with early‑stopping (no SumR improvement for 10 epochs).
• Loss weights: $\lambda_1=0.05$, $\lambda_2=1$, $\lambda_3=0.001$, $\lambda_4=0.004$.
• Proxy number: K=6 for $\mathcal{L}_{\mathrm{PM}}$.
• Retrieval score: Frame‑level score from CSA is summed with clip‑level score for final ranking.

---

Citation

If you find this repository useful, please cite:

@article{zhang2025enhancing,
  title={Enhancing Partially Relevant Video Retrieval with Robust Alignment Learning},
  author={Zhang, Long and Song, Peipei and Dong, Jianfeng and Li, Kun and Yang, Xun},
  journal={arXiv preprint arXiv:2509.01383},
  year={2025}
}

---

Acknowledgements

We build on prior open‑source implementations, including:

• MS‑SL
• GMMFormer
• GMMFormer v2
• UATVR

Thanks to the community for releasing datasets and features that make PRVR research possible.