2. Attach ECRD as a HuggingFace logits processor.

August 20, 2026 · View on GitHub

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs

Yongchang Zhang^1,3,* Oliver Ma^2,* Tianyi Liu^1,† Guangquan Zhou^1,† Yang Chen^1,†

¹Southeast University ²University of Oxford ³AIIA, Ministry of Education, China

* Equal contribution. † Corresponding author.

CVPR 2026

News

2026-06: See It, Say It, Sorted is published in the CVPR 2026 proceedings.
2026-02-25: The paper is released on arXiv.

Large vision-language models (LVLMs) can solve complex multimodal tasks with long chain-of-thought reasoning, yet their reasoning can gradually drift away from the visual evidence. Once an intermediate step becomes visually inconsistent, the error may propagate through an otherwise fluent reasoning trace and lead to an incorrect final answer.

We introduce Evidence-Constrained Reweighting Decoding (ECRD), a lightweight, training-free, plug-and-play framework for visually grounded multimodal reasoning. ECRD maintains a textual visual-evidence pool and uses it to supervise token selection during decoding. When the current evidence is insufficient to resolve an uncertain step, a lightweight visual decider extracts an additional piece of image-grounded micro-evidence and appends it to the pool. This iterative process keeps reasoning aligned with the image without SFT, RL, reward-model training, or modification of the LVLM parameters.

Across multiple LVLM backbones and benchmarks, ECRD substantially reduces hallucination while improving reasoning accuracy. The paper reports 16.5%–29.5% improvements on TreeBench and a 13.7% RH-AUC gain on RH-Bench, all without additional training.

Motivation

Reasoning pattern comparison

Conventional greedy decoding can amplify a single visually unsupported token into a wrong reasoning trajectory. ECRD instead introduces decoding-time visual supervision: existing evidence first reweights plausible next-token candidates, and the model consults the image again only when the evidence pool is insufficient.

Highlights

Training-free: no SFT, RL, reward model, or parameter update.
Plug-and-play: integrates with HuggingFace generation as a standard LogitsProcessor.
Evidence-constrained decoding: candidate tokens are reweighted using textual visual evidence at every reasoning step.
Dynamic evidence pool: newly extracted micro-evidence is cached and reused by later decoding steps.
Adaptive visual consultation: the visual decider is invoked only when uncertainty remains after evidence-based reweighting.
Backbone-friendly: the implementation supports Qwen2.5-VL and can be adapted to LLaVA-OneVision and InternVL-style LVLMs by changing only model-specific visual/chat preprocessing.

Main Results

ECRD improves visually grounded reasoning without additional model training.

Benchmark	Reported improvement
TreeBench	+16.5% to +29.5%
RH-Bench	+13.7% RH-AUC

The key result is not only higher final-answer accuracy, but also a reduction in reasoning-perception drift: the generated reasoning trace remains better aligned with the visual evidence as decoding proceeds.

Usage

Install

git clone https://github.com/uuuuZYC/ECRD.git
cd ECRD

pip install -e .

For the Qwen2.5-VL / GRIT visual-decider example, install the additional dependencies:

pip install -r requirements.txt

qwen-vl-utils is required only for the Qwen2.5-VL demo and GRITClient. The core ECRD supervisor (EvidenceScorer + ECRDLogitsProcessor) depends only on the standard project dependencies defined in pyproject.toml.

Minimal API Usage

from transformers import LogitsProcessorList
from ecrd import (
    Evidence,
    EvidenceScorer,
    ECRDLogitsProcessor,
    MixedGapTrigger,
    GRITClient,
)

# 1. Initialize the evidence pool with a global visual description.
scorer = EvidenceScorer(
    model=model,
    tokenizer=tokenizer,
    max_prefix_len=128,
)
scorer.add_evidence(
    Evidence(
        id="global-0",
        text=global_description,
        source="global",
        time_step=0,
    )
)

# 2. Attach ECRD as a HuggingFace logits processor.
ecrd = ECRDLogitsProcessor(
    scorer=scorer,
    tokenizer=tokenizer,
    min_k=1,
    max_k=64,
)

# 3. Optional: use GRIT as an uncertainty-triggered visual decider.
grit = GRITClient(model_id="yfan1997/GRIT-20-Qwen2.5-VL-3B")

def grit_hook(image, question, prefix_text, candidates):
    return grit.decide_next_token(
        image=image,
        question=question,
        prefix_text=prefix_text,
        candidates=candidates,
        max_new_tokens=64,
    )

ecrd.set_grit_runtime(
    hook=grit_hook,
    trigger=MixedGapTrigger(
        gap_thresh=0.08,
        min_k=2,
        cooldown=5,
    ),
    evidence_pool=scorer,
    question=question,
    image=image_uri,
)

# 4. Generate with ECRD.
outputs = model.generate(
    **inputs,
    logits_processor=LogitsProcessorList([ecrd]),
    do_sample=False,
    max_new_tokens=512,
)

Qwen2.5-VL Quick Demo

python examples/qwen2_5_vl_ecrd_demo.py \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --image /path/to/image.jpg \
  --question "What is shown in the image?" \
  --use-grit

For LLaVA-OneVision or InternVL-style models, the ECRD objects remain unchanged. Replace only the backbone-specific image and chat preprocessing before model.generate(...).

Method

ECRD consists of three lightweight decoding-time components.

1. Visual Evidence Pool

ECRD starts from short textual descriptions of image evidence. Instead of repeatedly re-processing all visual information at every token, these evidence sentences are stored in a reusable pool and provide persistent visual constraints throughout the reasoning process.

2. Distribution Supervisor

At each decoding step, ECRD:

computes the LVLM's original next-token distribution;
selects a compact candidate set using knee-based top-k selection;
scores candidate tokens against the current evidence pool;
reweights the candidate distribution using evidence-induced preferences; and
checks whether the resulting distribution is sufficiently confident.

This preserves the base model's language distribution while steering ambiguous decisions toward candidates that are better supported by the image evidence.

3. Visual Decider

If evidence-constrained reweighting still leaves the model uncertain, ECRD invokes a lightweight visual decider. The decider receives the image, current reasoning context, and candidate set, then returns:

a visually grounded token decision for the current step; and
a short micro-evidence sentence.

The new evidence is appended to the pool and can supervise later reasoning steps, allowing ECRD to see again only when necessary.

Repository Layout

.
├── ecrd/
│   ├── evidence.py            # Evidence representation and pool
│   ├── scorer.py              # Textual evidence scorer
│   ├── logits_processor.py    # ECRD distribution supervisor
│   ├── triggers.py            # Uncertainty trigger
│   ├── grit_client.py         # Optional GRIT visual decider
│   └── prompts.py             # Visual-evidence prompts
├── examples/
│   └── qwen2_5_vl_ecrd_demo.py
├── assets/
│   ├── fig1_reasoning_pattern.png
│   └── fig2_ecrd_overview.png
├── requirements.txt
├── pyproject.toml
├── CITATION.bib
└── LICENSE

Advanced configuration

The implementation supports the following optional environment variables:

Variable	Default	Meaning
`ECRD_MIN_K`	`1`	Minimum knee candidate-set size
`ECRD_MAX_K`	`64`	Maximum knee candidate-set size
`ECRD_LOGMIX_ALPHA`	`0.6`	Base mixing coefficient before adaptive override
`ECRD_ALPHA_ADAPT`	`1`	Use adaptive `alpha = p_top` when enabled
`ECRD_MIX_REWEIGHT`	`1`	Mass-match the evidence distribution to base candidate mass
`ECRD_DEBUG`	`0`	Print per-step base/evidence/mixed token diagnostics
`ECRD_DEBUG_TOP`	`10`	Number of tokens shown in debug output

Notes

ECRD is a decoding-time method and does not modify LVLM parameters.
The optional GRITClient is implemented for Qwen2.5-VL-style visual-decider usage; the distribution supervisor itself is a generic Transformers LogitsProcessor.
This repository focuses on the core ECRD implementation and intentionally omits project-specific experimental utilities, cached runs, and dataset-conversion code.

Citation

If you find this work useful, please cite:

@InProceedings{Zhang_2026_CVPR,
  author    = {Zhang, Yongchang and Ma, Oliver and Liu, Tianyi and Zhou, Guangquan and Chen, Yang},
  title     = {See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2026},
  pages     = {11933--11942}
}

License

This project is released under the MIT License.