AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation (ACL 2026)

August 12, 2026 · View on GitHub

MIT license arXiv

Authors: Rui Qian, Chuanhang Deng, Qiang Huang, Jian Xiong, Mingxuan Li, Yingbo Zhou, Wei Zhai, Jintao Chen, Dejing Dou†.

Abstract

Reasoning segmentation requires models to ground complex, implicit textual queries into precise pixel-level masks. Existing approaches rely on a single segmentation token , whose hidden state implicitly encodes both semantic reasoning and spatial localization, limiting the model's ability to explicitly disentangle what to segment from where to segment. We introduce AnchorSeg, which reformulates reasoning segmentation as a structured conditional generation process over image tokens, conditioned on language grounded query banks. Instead of compressing both semantic reasoning and spatial localization into a single embedding, AnchorSeg constructs an ordered sequence of query banks: latent reasoning tokens that capture intermediate semantic states, and a segmentation anchor token that provides explicit spatial grounding. We model spatial conditioning as a factorized distribution over image tokens, where the anchor query determines localization signals while contextual queries provide semantic modulation. To bridge token-level predictions and pixel-level supervision, we propose Token--Mask Cycle Consistency (TMCC), a bidirectional training objective that enforces alignment across resolutions. By explicitly decoupling spatial grounding from semantic reasoning through structured language grounded query banks, AnchorSeg achieves state-of-the-art results on ReasonSeg test set (67.7% gIoU and 68.1% cIoU). All code and models are publicly available at https://github.com/rui-qian/AnchorSeg.

News

  • [2026.4.7] AnchorSeg has been accepted to ACL 2026🎉🎉🎉!
  • [2026.8.12] AnchorSeg code and AnchorSeg-LLaVA-v1.5-7B models are released. Welcome to check them out!
  • [2026.4.18] Paper is released and GitHub repo is created.

🔥 Visual Grounding Family

Language to Grounding in vision and Beyond:

UGround: Towards Unified Visual Grounding with Unrolled Transformers (ICML 2026)
Authors: Rui Qian, Xin Yin, Chuanhang Deng, Zhiyuan Peng, Jian Xiong, Wei Zhai, Dejing Dou
github github arXiv Paper ICML Poster

AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation (ACL 2026)
Authors: Rui Qian, Chuanhang Deng, Qiang Huang, Jian Xiong, Mingxuan Li, Yingbo Zhou, Wei Zhai, Jintao Chen, Dejing Dou
github github arXiv Paper ACL Poster ACL Video ACL Slides

Reasoning to Attend: Try to Understand How Token Works (CVPR 2025)
Authors: Rui Qian, Xin Yin, Dejing Dou
github github arXiv Paper CVPR Poster

Installation Guide

#!/bin/bash
# 1. curl -O https://repo.anaconda.com/archive/Anaconda3-2025.06-0-Linux-x86_64.sh
# 2. bash Anaconda3-2025.06-0-Linux-x86_64.sh
# 3. conda create -n uground python=3.9
# 4. conda activate uground
# 5. chmod +x build.sh 
# 6. ./build.sh
# 7. wget https://github.com/Dao-AILab/flash-attention/releases/download/v2.6.3/flash_attn-2.6.3+cu118torch2.0cxx11abiFALSE-cp39-cp39-linux_x86_64.whl
# 8. pip install flash_attn-2.6.3+cu118torch2.0cxx11abiFALSE-cp39-cp39-linux_x86_64.whl
# 9. chmod +x install.sh
#10. ./install.sh

For ease of installation, we have encapsulated the setup steps into a script, build.sh. You can complete the environment configuration within 5 minutes.

Model and Dataset Preparation

Currently, we support 8 dataset types, namely: A: sem_seg, B: refer_seg, C: neg_refer_seg, D: correct_refer_seg, E: vqa, F: reason_seg, G: reason_seg_plus, and H: multi_reason_seg. Please Visit UGround dataset page for more details.

A: sem_seg: ade20k||cocostuff||pascal_part||paco_lvis||mapillary

B: refer_seg: refclef||refcoco||refcoco+||refcocog||refzom||grefcoco

C: neg_refer_seg: R-refcoco||R-refcoco+||R-refcocog

D: correct_refer_seg: fprefcoco||fprefcoco+||fprefcocog

E: vqa: llava_instruct_150k

F: reason_seg: ReasonSeg|train

G: reason_seg_plus(LISA++): instance_seg||cot||conversations||caption

H: multi_reason_seg(muse): MultiReasonSeg|train

Model NamegIoU cIoUSnapshotHG-ckpt URL
Results on ReasonSeg
AnchorSeg-LLaVA-v1.5-7B_ema/val67.20 75.15archiveweights
Results on gReferSeg
AnchorSeg-LLaVA-v1.5-7B74.76 68.68archiveweights

Experimental results

Training

./scripts/7b_reason_seg_val/train_anchorseg_llava1.5_ema.sh     # for ReasonSeg 7B
./scripts/13b_reason_seg_val/train_anchorseg_llava1.5_ema.sh    # for ReasonSeg 13B

Merge LoRA Weight

./scripts/7b_reason_seg_val/merge_lora_weight_uground_llava1.5_ema.sh     # for ReasonSeg 7B
./scripts/13b_reason_seg_val/merge_lora_weight_uground_llava1.5_ema.sh    # for ReasonSeg 13B

Validation

./scripts/7b_reason_seg_val/eval_anchorseg_llava1.5_ema.sh     # for ReasonSeg 7B
./scripts/13b_reason_seg_val/eval_anchorseg_llava1.5_ema.sh    # for ReasonSeg 13B

Acknowledgements

We are grateful for the foundational code provided by PixelLM, SESAME, GSVA, READ, LISA, LLaVA, and SAM. Utilizing their resources implies agreement to their respective licenses. Our project benefits greatly from these contributions, and we acknowledge their significant impact on our work.

Citation

If you use our work or our implementation in this repo, or find them helpful, please consider giving a citation.

@inproceedings{qian2026UGround,
  title={UGround: Towards Unified Visual Grounding with Unrolled Transformers},
  author={Qian, Rui and Yin, Xin and Deng, Chuanhang and Peng, Zhiyuan and Xiong, Jian and Zhai, Wei and Dou, Dejing},
  booktitle={International Conference on Machine Learning},
  year={2026}
}
@inproceedings{qian2026AnchorSeg,
  title={AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation},
  author={Qian, Rui and Deng, Chuanhang and Huang, Qiang and Xiong, Jian and Li, Mingxuan and Zhou, Yingbo and Zhai, Wei and Chen, Jintao and Dou, Dejing},
  booktitle={Annual Meeting of the Association for Computational Linguistics},
  year={2026}
}
@inproceedings{qian2025reasoning,
  title={Reasoning to Attend: Try to Understand How <SEG> Token Works},
  author={Qian, Rui and Yin, Xin and Dou, Dejing},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  year={2025}
}

Contact

If you have any questions, feel free to reach out at qiianruii@gmail.com, dengch2000@gmail.com, and dejingdou@gmail.com.