Omni-Referring Image Segmentation

🗓️TODO

• Release training codes and OmniRef dataset.
• Release paper.

🛠️ Installation

• Create a conda virtual environment and activate it

conda create -n omnisegnet python=3.8 -y
conda activate omnisegnet

• Install Pytorch following the official installation instructions

# CUDA 11.3
pip3 install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

• Install Detectron following the official installation instructions

python -m pip install 'git+https://github.com/MaureenZOU/detectron2-xyz.git'

• Compile the MSDeformAttn layer:

cd OmniSegNet_model/modeling/pixel_decoder/ops
sh make.sh

pip install -r requirements.txt

wget https://github.com/explosion/spacy-models/releases/download/en_vectors_web_lg-2.1.0/en_vectors_web_lg-2.1.0.tar.gz -O en_vectors_web_lg-2.1.0.tar.gz
pip install en_vectors_web_lg-2.1.0.tar.gz
pip install albumentations
pip install Pillow==9.5.0
pip install tensorboardX

📚Data Preparation

• The data structure should look like the following:

| -- datasets
        | -- anns
            | -- OmniRef
                | -- OmniRef.json
                | -- instances.json
            | -- gRefCOCO
                | -- grefs(unc).json
                | -- instances.json
        | -- images
            | -- train2014
                | -- COCO_train2014_XXXXX.jpg
                | -- ...
  

🚀Training

Firstly, download the backbone weights (swin_base_patch4_window12_384_22k.pth) and (bert-base-uncased).

sh train.sh

🤝 Acknowledgments

This project is based on refer, ReLA, Detectron2, VRP-SAM. Many thanks to the authors for their great works! 

✏️ Citation

If you find our paper and code helpful, we kindly invite you to give it a star and consider citing our work.

@article{zheng2025omni,
  title={Omni-Referring Image Segmentation},
  author={Zheng, Qiancheng and Shen, Yunhang and Luo, Gen and Song, Baiyang and Sun, Xing and Sun, Xiaoshuai and Zhou, Yiyi and Ji, Rongrong},
  journal={arXiv preprint arXiv:2512.06862},
  year={2025}
}