Data Preparation (Same as SgMg)

After the organization, we expect the directory structure to be the following:

Notice: SgMg and datasets are in the same folder.

├── SgMg
├── datasets
│   ├── coco
│       ├── train2014
│       ├── refcoco
│           ├── instances_refcoco_train.json
│           ├── instances_refcoco_val.json
│       ├── refcoco+
│           ├── instances_refcoco+_train.json
│           ├── instances_refcoco+_val.json
│       ├── refcocog
│           ├── instances_refcocog_train.json
│           ├── instances_refcocog_val.json
│   ├── refer_youtube_vos
│         ├── meta_expressions
│         ├── train
│               ├── JPEGImages
│               ├── Annotations
│               ├── meta.json
│         ├── valid
│               ├── JPEGImages
│   ├── refer_davis
│         ├── meta_expressions
│         ├── valid
│               ├── JPEGImages
│                     ├── 480p
│               ├── Annotations
│               ├── ImageSets
│               ├── meta.json
│   ├── a2d_sentences
│       ├── Release
│       ├── text_annotations
│           ├── a2d_annotation_with_instances
│           ├── a2d_annotation.txt
│           ├── a2d_missed_videos.txt
│       ├── a2d_sentences_single_frame_test_annotations.json
│       ├── a2d_sentences_single_frame_train_annotations.json
│       ├── a2d_sentences_test_annotations_in_coco_format.json
│   ├── jhmdb_sentences
│       ├── Rename_Images
│       ├── puppet_mask
│       ├── jhmdb_annotation.txt
│       ├── jhmdb_sentences_samples_metadata.json
│       ├── jhmdb_sentences_gt_annotations_in_coco_format.json
...

Ref-COCO

Download the dataset from the official website COCO.
RefCOCO/+/g use the COCO2014 train split. Download the annotation files from github.

Convert the annotation files:

python3 tools/data/convert_refexp_to_coco.py

Finally, we expect the directory structure to be the following:

├── datasets
│   ├── coco
│       ├── train2014
│       ├── refcoco
│           ├── instances_refcoco_train.json
│           ├── instances_refcoco_val.json
│       ├── refcoco+
│           ├── instances_refcoco+_train.json
│           ├── instances_refcoco+_val.json
│       ├── refcocog
│           ├── instances_refcocog_train.json
│           ├── instances_refcocog_val.json

refer_youtube_vos

Download the dataset from the competition's website here. Then, extract and organize the file. We expect the directory structure to be the following:

├── datasets
│   ├── refer_youtube_vos
│         ├── meta_expressions
│         ├── train
│               ├── JPEGImages
│               ├── Annotations
│               ├── meta.json
│         ├── valid
│               ├── JPEGImages

refer_davis17

Notice: We recommend to directly download the parsed Ref-DAVIS dataset from the Google Drive to avoid the following steps.

Download the DAVIS2017 dataset from the website. Note that you only need to download the two zip files DAVIS-2017-Unsupervised-trainval-480p.zip and DAVIS-2017_semantics-480p.zip. Download the text annotations from the website. Then, put the zip files in the directory as follows.

├── datasets
│   ├── refer_davis
│   │   ├── DAVIS-2017_semantics-480p.zip
│   │   ├── DAVIS-2017-Unsupervised-trainval-480p.zip
│   │   ├── davis_text_annotations.zip

Unzip these zip files.

unzip -o davis_text_annotations.zip
unzip -o DAVIS-2017_semantics-480p.zip
unzip -o DAVIS-2017-Unsupervised-trainval-480p.zip

Preprocess the dataset to refer_youtube_vos format. (Make sure you are in the main directory)

python tools/data/convert_davis_to_ytvos.py

Finally, unzip the file DAVIS-2017-Unsupervised-trainval-480p.zip again (since we use mv in preprocess for efficiency).

unzip -o DAVIS-2017-Unsupervised-trainval-480p.zip

A2D-Sentences

Follow the instructions and download the dataset from the website here. Then, extract the files. Additionally, we use the same json annotation files generated by MTTR. Please download these files from google drive. We expect the directory structure to be the following:

├── datasets
│   ├── a2d_sentences
│   │   ├── Release
│   │   ├── text_annotations
│   │   │   ├── a2d_annotation_with_instances
│   │   │   ├── a2d_annotation.txt
│   │   │   ├── a2d_missed_videos.txt
│   │   ├── a2d_sentences_single_frame_test_annotations.json
│   │   ├── a2d_sentences_single_frame_train_annotations.json
│   │   ├── a2d_sentences_test_annotations_in_coco_format.json

JHMDB-Sentences

Follow the instructions and download the dataset from the website here. Then, extract the files. Additionally, we use the same json annotation files generated by MTTR. Please download these files from google drive. We expect the directory structure to be the following:

├── datasets
│   ├── jhmdb_sentences
│   │   ├── Rename_Images
│   │   ├── puppet_mask
│   │   ├── jhmdb_annotation.txt
│   │   ├── jhmdb_sentences_samples_metadata.json
│   │   ├── jhmdb_sentences_gt_annotations_in_coco_format.json