---

🔥 News

• [2025-12-31] 🚀 Code and paper are released!
• [2025-12-XX] 🚧 We are preparing the demo on HuggingFace.

---

💡 Abstract

Reasoning Segmentation requires models to interpret complex linguistic queries for pixel-level localization. While current SFT and RL methods suffer from catastrophic forgetting and training instability, we propose Evol-SAM3, a novel zero-shot framework.

Instead of a static "generate-then-segment" paradigm, we model the task as an Evolutionary Search process:

1. Dynamic Evolution: We maintain a population of prompts and refine them via a "Generate-Evaluate-Evolve" loop.
2. Visual Arena: A tournament-based selection mechanism using MLLMs to assess mask quality without ground truth.
3. Semantic Mutation: Injecting diversity and correcting hallucinations during inference.
4. Heterogeneous Arbitration: A final safeguard combining text-based reasoning with geometric intuition.

Evol-SAM3 significantly outperforms static baselines (e.g., SAM3 Agent) and even fully supervised SOTA methods (e.g., LISA-13B) on ReasonSeg and RefCOCO benchmarks.

---

🛠️ Methodology

Our framework consists of three phases:

• Phase 1: Initialization. A meta-generator expands the query into diverse hypotheses.
• Phase 2: Evolutionary Loop. Prompts compete in a Visual Arena, and winners undergo Semantic Mutation to breed better generations.
• Phase 3: Final Arbitration. A double-blind swap mechanism selects the best mask between evolutionary results and geometric priors.

---

📊 Performance

We conduct extensive experiments on ReasonSeg and RefCOCO series benchmarks. Evol-SAM3 achieves superior performance without any parameter updates.

🏆 Comparison on ReasonSeg (Zero-Shot vs. SFT)

Evol-SAM3 (7B) outperforms not only other training-free agents but also fully supervised SOTA methods (e.g., LISA-13B).

Method	Type	Backbone	Val gIoU	Test gIoU
LISA [CVPR'24]	SFT	LLaVA-1.5 13B	65.0	61.3
GLaMM [CVPR'24]	SFT	Vicuna 7B	47.4	--
SAM 3 Agent [arXiv'25]	Training-free	Qwen2.5-VL 72B	74.6	70.8
RSVP [arXiv'25]	Training-free	GPT-4o	64.7	60.3
Evol-SAM3 (Ours)	Training-free	Qwen2.5-VL 7B	70.7	72.5

Note: Our 7B model surpasses the 72B baseline on the challenging Test set, proving the efficiency of evolutionary search.

📈 Comparison on Referring Expression Segmentation (Zero-Shot)

Comparison with state-of-the-art zero-shot methods on RefCOCO/+/g.

Method	Backbone	RefCOCO (val)	RefCOCO+ (val)	RefCOCOg (val-U)
SAM 3 Agent	Qwen2.5-VL 7B	59.4	51.4	57.2
Evol-SAM3 (Ours)	Qwen2.5-VL 7B	68.7	64.4	64.7
Improvement	--	+9.3	+13.0	+7.5

All results are reported in cIoU. Evol-SAM3 significantly narrows the gap with supervised methods.

---

🖼️ Qualitative Results

Comparison between SAM3 Agent (Baseline) and Evol-SAM3 (Ours). Our method successfully handles functional descriptions and corrects visual biases.

---

🚀 Quick Start

1. Environment Setup

Create and activate the conda environment:

conda env create -f Evol-SAM3.yml
conda activate Evol-SAM3

2. Data Preparation

Download Datasets:

• For dataset, you can download it from here.

Organize your datasets in the DATASET directory as follows:

DATASET/
├── reason_seg/
│   └── ReasonSeg/
│       ├── train/
│       ├── val/
│       └── test/
└── refer_seg/
    ├── images/
    │   └── train2014/
    ├── refcoco/
    ├── refcoco+/
    └── refcocog/

3. Model Weights

Download the required model weights:

• SAM3 Checkpoint: Download sam3.pt from Hugging Face (requires access request).
• MLLM Checkpoint: The MLLM directory structure should look like this:

MLLM/
├── Qwen2.5_VL_3B/
└── Qwen2.5_VL_7B/
    └── Qwen/
        └── Qwen2___5-VL-7B-Instruct/
            ├── config.json
            ├── model.safetensors
            ├── tokenizer.json
            └── ...

Download Qwen2.5-VL-7B-Instruct .

4. Configuration

Update the configuration file configs/ReasonSeg_7B.yaml to point to the correct paths:

paths:
  qwen_model_path: "MLLM/Qwen2.5_VL_7B/Qwen/Qwen2___5-VL-7B-Instruct"
  sam3_ckpt_path: "sam3/sam3.pt"
  dataset_root: "DATASET/reason_seg/ReasonSeg"
  log_dir: "logs/ReasonSeg_7B"

5. Inference

Run the inference script:

bash ReasonSeg_7B.sh

This script will execute:

python main.py --config configs/ReasonSeg_7B.yaml

Resume Inference: If the inference is interrupted, you can resume it by specifying the log directory with the --resume argument:

python main.py --config configs/ReasonSeg_7B.yaml --resume logs/ReasonSeg_7B/ReasonSeg_7B_xxxxx

(Replace ReasonSeg_7B_xxxxx with your actual log directory name)

📝 Citation

If you find our work helpful, please consider citing:

@misc{ye2025evolvingtrainingzeroshotreasoning,
      title={Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting}, 
      author={Kai Ye and Xiaotong You and Jianghang Lin and Jiayi Ji and Pingyang Dai and Liujuan Cao},
      year={2025},
      eprint={2512.24702},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.24702}, 
}