:stars:SPaTS

August 1, 2026 · View on GitHub

One Patch is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-based Scene Text Spotting

ACM International Conference on Multimedia (ACM MM), 2026

:ocean:Introduction

SPaTS (Single-Patch Text Spotting) is a vision-centric MLLM framework for scene text spotting. Unlike prior methods that assign multiple patches per text instance, SPaTS represents each instance with a single optimal anchor patch, eliminating redundancy and localization ambiguity.

SPaTS overview

Overview of SPaTS: (a) architecture, (b) two-stage training, (c) DEA, and (d) PED.

SPaTS introduces three core components:

Component	Full Name	Description
SPaSO	Single-patch Selective Optimization	RL-based visual-token selection via patch-level rewards (GRPO)
DEA	Directional Embedding Alignment	Decouples feature magnitude and direction for stable visual prototypes
PED	Patch-Enhanced Decoding	Fuses selected patch with language hidden states for precise geometry regression

:earth_asia:Environment

Requirements: Python 3.11 · CUDA 12 · PyTorch 2.8

git clone https://github.com/eeNickTang/spats.git
cd spats
conda create -n spats python=3.11
conda activate spats

pip install -e .

# Required by PED training and batch inference.
pip install flash-attn --no-build-isolation

Paper-default hyperparameters live in configs/sft/ and configs/spaso/. W&B is disabled by default. To enable it, run wandb login, set report_to: wandb, and adjust wandb_project / wandb_group in the YAML.

:package:Model Weights

Reproduce from public Qwen3-VL bases:

Model	HuggingFace ID
2B base	`Qwen/Qwen3-VL-2B-Instruct`
4B base	`Qwen/Qwen3-VL-4B-Instruct`

Full pipeline (SFT → SPaSO) and checkpoint paths: configs/README.md.

:hammer_and_pick:Data Preparation

Training data is the same as SPTS. Download and organize datasets following the Dataset section in the SPTS README, then place them under data/origin/:

data/origin
├─CTW1500
│  ├─annotations
│  │      test_ctw1500_maxlen25.json
│  │      train_ctw1500_maxlen25_v2.json
│  ├─ctwtest_text_image
│  └─ctwtrain_text_image
├─icdar2013
│  │  ic13_test.json
│  │  ic13_train.json
│  ├─test_images
│  └─train_images
├─icdar2015
│  │  ic15_test.json
│  │  ic15_train.json
│  ├─test_images
│  └─train_images
├─mlt2017
│  │  train.json
│  └─MLT_train_images
├─syntext1
│  │  train.json
│  └─syntext_word_eng
├─syntext2
│  │  train.json
│  └─emcs_imgs
└─totaltext
    │  test.json
    │  train.json
    ├─test_images
    └─train_images

Convert SPTS-format data to SPaTS training JSONL:

bash dataset/scripts/spats_preprocess.sh

Or call the preprocessor directly:

python dataset/spats_preprocess.py \
    --data_root data/origin \
    --out_root data/SPaTS

The preprocessor writes each available dataset split in the same format:

data/SPaTS/spot/<dataset>/<split>/
├── annotations.jsonl
└── images/

Both train/ and test/: totaltext, CTW1500, icdar2013, icdar2015
train/ only: mlt2017, syntext1, syntext2

SFT reads the train/ splits, while inference and evaluation read the test/ splits. In training configs, dataset_root is the parent directory data/SPaTS; the configured task (task: spot) selects its spot/ subdirectory. The generated data/SPaTS/ directory is ignored by Git and must be created on each machine.

:rocket:Train

Training follows a two-stage pipeline starting from Qwen3-VL base weights (not included). Hyperparameters are under configs/sft/ and configs/spaso/; shell scripts only launch distributed jobs.

Stage 1 — Supervised Fine-Tuning (SFT) — initializes from Qwen/Qwen3-VL-*-Instruct

bash run_scripts/sft/spot_sft_2b.sh   # configs/sft/sft_2b.yaml
bash run_scripts/sft/spot_sft_4b.sh   # configs/sft/sft_4b.yaml

Stage 2 — SPaSO (Reinforcement Learning) — initializes from SFT checkpoint

bash run_scripts/spaso/spot_spaso_2b.sh   # configs/spaso/spaso_2b.yaml
bash run_scripts/spaso/spot_spaso_4b.sh   # configs/spaso/spaso_4b.yaml

The scripts default to 8 GPUs. Override GPU count with NPROC; pass a custom config as the first argument:

NPROC=4 CUDA_VISIBLE_DEVICES=0,1,2,3 \
  bash run_scripts/sft/spot_sft_4b.sh configs/sft/sft_4b.yaml

:clinking_glasses:Inference

Batch inference over test splits:

bash inference/scripts/SPaTS/infer.sh \
  checkpoints/SPaTS-4B-GRPO all SPaTS-4B-GRPO

Arguments are MODEL_PATH [DATASETS] [MODEL_NAME]. Predictions are saved under output/inference/{bbox,point,bezier}/SPaTS-4B-GRPO-all/.

Single-image demo:

python inference/demo/infer.py \
    --model_path checkpoints/SPaTS-4B-GRPO \
    --image_path /path/to/image.jpg

:bar_chart:Evaluation

Run inference first, then pass the same model name and dataset selection:

bash eval/scripts/eval_spot_bbox.sh SPaTS-4B-GRPO all
bash eval/scripts/eval_spot_bezier.sh SPaTS-4B-GRPO all
bash eval/scripts/eval_spot_point.sh SPaTS-4B-GRPO all
bash eval/scripts/eval_spot_patch.sh SPaTS-4B-GRPO all

Results are written under output/eval/spot/.

Lexicon-based evaluation uses the same auxiliary ground-truth and lexicon files as SPTS. Follow the Evaluation section of the SPTS README, download both packages, and organize them as follows:

eval/data/
├── label/
│   ├── gt_ctw1500/
│   ├── gt_ic13/
│   ├── gt_ic15/
│   └── gt_totaltext/
└── lexicons/
    ├── ctw1500/
    ├── ic13/
    ├── ic15/
    └── totaltext/

The evaluation launchers use eval/data/ by default. These auxiliary files are not included in this repository and are required for the default lexicon-based evaluation. Raw data, generated data/SPaTS/, checkpoints, and evaluation outputs are also excluded from Git.

:mag:Case

SPaTS case study.

:clipboard:Citation

If you find our work useful or interesting, please feel free to cite it!

@misc{tang2026onepatch,
  title         = {One Patch Is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-Based Scene Text Spotting},
  author        = {Tang, Rui and Yang, Wentao and Zhang, Peirong and Shi, Yongxin and Zhang, Shun and He, Huiguo and Jin, Lianwen},
  year          = {2026},
  eprint        = {2607.27902},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2607.27902}
}