conda create --name cure
conda activate cure

To install the necessary packages, you can use the provided requirements.txt file:

python -m pip install -r requirements.txt

or

conda env create -f environment.yml

The code was tested on Python 3.10 and PyTorch 2.4.

WebVid-CoVR

To use the WebVid-CoVR dataset, you will have to download the WebVid videos and merge annotation.

To merge the annotation

cd annotation/webvid-covr2m/
cat all_csvs.zip.* > all_csvs.zip
unzip all_csvs.zip

To download the videos, install mpi4py (conda install -c conda-forge mpi4py) and run:

ln -s /path/to/your/datasets/folder datasets
python tools/scripts/download_covr.py

Before training, you will need to compute the BLIP embeddings for the videos/images. To do so, run:

# This will compute the BLIP embeddings for the WebVid-CoVR videos. 
python tools/embs/save_blip_embs_vids.py --video_dir datasets/WebVid/2M --todo_ids annotation/webvid-covr2m/webvid2m-covr_train.csv 

# This will compute the BLIP embeddings for the Test videos.
python tools/embs/save_blip_embs_vids.py --video_dir datasets/WebVid/2M --todo_ids annotation/webvid-covr2m/webvid2m-covr_test.csv 

# This will compute the Multimodal BLIP embeddings for the WebVid-CoVR videos. 
python tools/embs/save_blip_embs_vids_multimodal.py --video_dir datasets/WebVid/2M --todo_ids annotation/webvid-covr2m/webvid2m-covr_train.csv 

The command to launch a training experiment is the folowing:

python train.py [OPTIONS]

The parsing is done by using the powerful Hydra library. You can override anything in the configuration by passing arguments like foo=value or foo.bar=value. See Options parameters section at the end of this README for more details.

The command to evaluate is the folowing:

python test.py test=<test> [OPTIONS]