This is the PyTorch Implementation of PSRN

Introduction

This repository is Pytorch implementation of Progressive Semantic Reconstruction Network for Weakly Supervised Referring Expression Grounding.

Prerequisites

• Python 3.6
• Pytorch 1.9.0
• CUDA 11.1

Installation

1. Please refer to MattNet to install mask-faster-rcnn, REFER and refer-parser2.
   Follow Step 1 & 2 in Training to prepare the data and features. Place the proposal features extracted by mrcn in the directory cache/feats.

2. Download the data files needed for training, including Glove, semantic similarity and subject and object word. Place these files uniformly in cache. You can also follow the steps below to generate these data yourself:

python tools/gen_wds.py --dataset ${DATASET} --splitBy ${SPLITBY}  
python tools/cal_sim.py --dataset ${DATASET} --splitBy ${SPLITBY}  

Generating Key Triads

Generate the Key Triads using Spacy:

python tools/gen_triads.py --dataset ${DATASET} --splitBy ${SPLITBY}   

You can also download the generated key triads file.

Training

Train PSRN :

python tools/train.py --dataset refcocog --splitBy google --stage 1  

You can change --resume to load the staged trained weights file for different modes of training.

Evaluation

Evaluate PSRN with trained weights:

python tools/eval.py --dataset refcocog --splitBy google --split val --stage 3

The trained weights file can be downloaded from here, place them in the directory output/.

Acknowledge

This code is partially based on DTWREG and MAttNet.