README.md

Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning

If you like our project, please give us a star ⭐ on GitHub for the latest update.

Latest Updates :loudspeaker:

• [2025/06/09] Data for training and evaluation are released in our huggingface repo.🤗
• [2025/06/09] Code are available now! Welcome to watch 👀 this repository for the latest updates.

Overview 👀

Large Vision-Language Models (LVLMs) have demonstrated impressive general capabilities across a wide range of multi-modal tasks. However, the reasoning processes of LVLMs often suffer from unreliable outputs and limited interpretability. To address this, grounded visual reasoning has emerged as a promising paradigm that enforces responses anchored on salient visual evidence regions. However, existing approaches typically rely on costly supervision such as bounding box annotations, chain-of-thought rationale or external tool calls, limiting their scalability. In this work, we propose Ground-R1, a reinforcement learning framework that enables grounded visual reasoning without requiring explicit evidence or rationale annotations. Ground-R1 consists of a grounding phase that generates evidence region rollouts based on format constraints, and an answering phase that produces responses guided by both answer correctness and format adherence rewards. Extensive experiments across multiple visual reasoning benchmarks manifest that Ground-R1 achieves superior performance and exhibits emergent cognitive behaviors such as uncertainty awareness, spatial perception, and iterative refinement, offering a scalable and interpretable alternative to existing approaches.

• We propose Ground-R1, a novel RL framework for grounded visual reasoning that eliminates the need for costly bounding box annotations, CoT rationales, and external tool calls.
• Ground-R1 decouples evidence region generation from answer synthesis, enabling interpretable reasoning via format-constrained grounding and reward-driven response generation.
• Extensive experiments across multiple benchmarks demonstrate that Ground-R1 achieves superior performance and exhibits emergent cognitive behaviors such as uncertainty awareness, spatial perception and iterative refinement.

Reasoning trajectories of Ground-R1 🔥

Performance 🏆

Evaluation results on the test split of VisCoT benchmark including doc/text and chart understanding and general VQA, relation reasoning, and fine-grained VQA (FGVQA).

Ground-R1 demonstrates substantial performance advantages over contemporary approaches: 1) Ground-R1 outperforms the baseline model Qwen2.5-VL-7B by $9.2%$on the average accuracy; 2) Compared to the recent grounded visual reasoning approach [CogCoM](https://github.com/THUDM/CogCoM), it achieves absolute improvements of$+24.2%$on TextVQA and$+21.0%$ on GQA; 3) Further more, our Ground-R1 surpasses recent R1-series works (e.g., R1-Onevision, Vision-R1-7B and LMM-R1, empirically validating that introducing evidence grounding during reasoning processes effectively enhances multi-modal task-solving capabilities.

To ensure comprehensive evaluation, we conduct systematic benchmarking across established LVLM benchmarks including MME, MM-Vet, SEED-Bench, MME-RealWorld-Lite (MME-RWL) , and RealworldQA for general capability evaluation and POPE for hallucination validation.

Ground-R1 demonstrates significant advantages over the baseline (Qwen2.5-VL-7B), existing grounded reasoning approaches (CogCoM, Sys2-LLaVA, VisCoT), and R1-series methods (R1-Onevision, Vision-R1-7B, LMM-R1). For instance, on the MME benchmark, Ground-R1 achieves performance improvements of $33.9 and \54.7$ points compared to Qwen2.5-VL-7B and Vision-R1-7B, respectively.

Besides, We evaluate the visual grounding capabilities of Ground-R1 on RefCOCO, RefCOCO+ and RefCOCOg.

Ground-R1 demonstrates superior visual grounding capabilities, achieving $92.9%$ accuracy on RefCOCO val and surpassing most LVLMs, while narrowing the performance gap between generalist and specialist models (e.g., Grounding DINO).

RL Training Curves 📈

As shown, the format reward converges rapidly during the early training stages, while the accuracy reward stabilizes after approximately 600 steps. Additionally, the response length gradually increases with training steps, indicating the emergence of longer reasoning chains.

The accuracy of the evidence regions generated by Ground-R1 ($\text{Bounding Box GIoU Score} = \frac{\text{GIoU} + 1}{2}$) steadily improves throughout training, reaching approximately $67.5%$, despite the absence of bounding box supervision in the RL training process.

Installation 🛠️

git clone https://github.com/zzzhhzzz/Ground-R1.git
cd Ground-R1
conda create --name ground-r1 python=3.10
conda activate ground-r1
bash setup.sh

Please download and unzip Visual-CoT dataset for training and evaluation. Download and unzip LVLM benchmarks and RefCOCO for evaluation.

Training 🚀

Add the path of the VisCoT folder to scripts/run_grpo_video.sh, then,

cd Ground-R1
bash scripts/run_grpo_video.sh

Inference & Evaluation 🔮

VisCoT Benchmark:

Please add your api_key and api_base to eval/multi_baseline.sh and eval/multi_Ground_R1.sh

cd Ground-R1
bash eval/multi_baseline.sh # For baseline model
bash eval/multi_Ground_R1.sh # For trained Ground-R1

LVLM Benchmarks:

bash eval_general_dataset/multi_Ground_R1.sh

Grounding Benchmarks:

bash eval/multi_RefCOCO.sh

Ablation 📐

SFT Training:

1. Download Vanilla_SFT.json and Ground_SFT.json from repo then put them to qwen-vl-finetune\qwenvl\data

2. Please add your dataset path (VisCoT) to qwen-vl-finetune/scripts/prepare_SFT_data.sh, then,

cd Ground-R1
bash qwen-vl-finetune/scripts/prepare_SFT_data.sh

Note: You can change --max_steps in Vanilla_SFT.sh and Ground_SFT.sh for a fair comparison, since Ground-R1 training only requires 8,000 samples.

cd qwen-vl-finetune
bash scripts/Vanilla_SFT.sh # For Vanilla SFT Training
bash scripts/Ground_SFT.sh # For Ground SFT Training

SFT Evaluation:

cd Ground-R1
### VisCoT Benchmark
bash eval/multi_baseline.sh # change --model_path for Vanilla SFT Eval
bash eval/multi_Ground_SFT.sh # For Ground SFT Eval
### LVLM Benchmarks
bash eval_general_dataset/multi_SFT.sh

Acknowledgement 👍

Our code is built upon Video-R1, thanks for their excellent works!

Citation ✏️

If you find the code and paper useful for your research, please consider staring this repo and citing our paper:

@article{cao2025ground,
  title={Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning},
  author={Cao, Meng and Zhao, Haoze and Zhang, Can and Chang, Xiaojun and Reid, Ian and Liang, Xiaodan},
  journal={arXiv preprint arXiv:2505.20272},
  year={2025}
}