Referral Video Object Segmentation with RefAM

🔧 Installation

Prerequisites

• Python 3.10+
• CUDA 12.1+
• Conda (recommended)

Setup

Initial conda env from RIOS.
Simple evaluation based on PA (point accuracy) can be run without SAM2 and without MUTR.
Install SAM2 to YOUR_SAM_PATH.

conda activate refam-env
cd refam/RVOS
pip install pycocotools

Download SAM checkpoint

wget https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt -O checkpoints/sam2.1_hiera_large.pt

For DAVIS Evaluation

Clone MUTR to YOUR_MUTR_PATH and compile, should work in the current evn. Ref-YouTube-VOS and MeViS can be evaluated completely without MUTR.

Data

Please refer to data.md for data preparation.

🚀 Quick Start

RVOS Notebook

Notebook

RVOS Scripts

sh scripts/davis.sh
sh scripts/youtube.sh
sh scripts/mevis.sh

RVOS Parameters

Processing pipeline:

1. Extract attention maps and find argmax with Mochi
2. [Intermediate step] Compute PA (point accuracy)
3. Compute segmentation maps with SAM2
4. Evaluate segmentation maps

Each step can be run independently of the previous.
Set the following params: YOUR_SAM_PATH, and YOUR_MUTR_PATH.

# 1 step [can be run multiple instances in parallel, videos are reshuffled]

python main.py \
--dataset davis \
--name_exp test \ # name for the output attention maps and intermediate steps
--mochi \ # preprocess with mochi
--compute_concepts \  # specify which attention maps to use
--filter_stop_words \  # filtering of attention maps
--general_caption_as_prompt \  # use captions as conditioning for mochi
--use_spacy \  # detect noun phrase
--use_spacy_direction  # detect directions in the ref.expression


# 2 step
python main.py --dataset davis --name_exp test --point_eval


# 3 step [can be run multiple instances in parallel, videos are reshuffled]
python main.py --dataset davis --name_exp test --sam2 --sam2_path YOUR_SAM_PATH


# 4 step 
python main.py --dataset davis --name_exp test --eval --eval_davis_path YOUR_MUTR_PATH


# steps 1 -> 3 -> 4
python main.py \
--dataset davis \
--name_exp test \ 
--mochi \ 
--compute_concepts \  
--filter_stop_words \ 
--general_caption_as_prompt \  
--use_spacy \  
--use_spacy_direction \
--sam2 \
--sam2_path YOUR_SAM_PATH \
--eval \
--eval_davis_path YOUR_MUTR_PATH

Reproducibility note

We note that the provided scripts use a higher number of stop words for RVOS than stated in the paper. Due to updates in the local cluster, the results with five stop words were not reproducible. However, subsequent experiments showed that the reported performance was matched when using the increased number of stop words.

Exact packages that were used in the environment can be seen here.