Fine-tuned checkpoint

Generation Enhances Understanding in Unified Multimodal Models
via Multi-Representation Generation

Zihan Su^1,2†*, Hongyang Wei^1*, Kangrui Cen^3*, Yong Wang^2‡, Guanhua Chen⁴,
Chun Yuan¹, Xiangxiang Chu²

¹ Tsinghua University ² AMAP, Alibaba Group
³ Shanghai Jiao Tong University ⁴ South China University of Technology
†Work done during internship at AMAP, Alibaba Group *Equal contribution ‡Project leadAuthor

Release

• [05/25] 🚀 🚀 Code Released!
• [01/30] Initial Preview Release 🔥 Coming Soon!

🔆 Introduction

We propose UniMRG, a simple yet effective architecture-agnostic post-training method for UMMs that leverages generation capabilities to enhance understanding.

💻 Installation

Our code builds on Harmon. Run the commands below from the Harmon directory.

To install requirements:

conda create -n harmon python=3.10
conda activate harmon
pip install -r requirements.txt

📋 File Preparation

Data Preparation

We provide a preprocessed dataset: LLaVA-Instruct-150K-UniMRG. Expected layout:

data/
├── LLaVA-Instruct-150K/
│   └── llava_v1_5_mix665k.json
│   └── tuning_data
│   └── tuning_data_depth
│   └── tuning_data_mask

You can use this preprocessed dataset directly, or build the data yourself with the pipeline below.

Download the LLaVA dataset from here. Expected layout:

data/
├── LLaVA-Instruct-150K/
│   └── llava_v1_5_mix665k.json
│   └── tuning_data/
│       ├── coco/
│       ├── gqa/
│       ├── ocr_vqa/
│       ├── textvqa/
│       └── vg/

Clone Depth-Anything-V2 and run the following command to produce depth maps:

python run.py \
  --encoder vitl \
  --img-path data/LLaVA-Instruct-150K/tuning_data \
  --outdir data/LLaVA-Instruct-150K/tuning_data_depth \
  --grayscale \
  --pred-only

Segmentation masks follow the Segment Anything automatic_mask_generator. Clone the original repository, copy segment-anything/run_mask.py from this repo into the root of that clone, then run:

python run_mask.py \
  --img-path data/LLaVA-Instruct-150K/tuning_data \
  --outdir data/LLaVA-Instruct-150K/tuning_data_mask \
  --checkpoint sam_vit_h_4b8939.pth \
  --model-type vit_h \
  --pred-only

Model Preparation

Download the pre-trained Harmon model:

# Create checkpoints directory
mkdir -p checkpoints

# Download via Hugging Face CLI (recommended)
pip install -U "huggingface_hub[cli]"
huggingface-cli download wusize/harmon --local-dir checkpoints --repo-type model

Your checkpoint structure should look like:

checkpoints/
├── kl16.ckpt
├── harmon_0.5b.pth
├── harmon_1.5b.pth
└── ...

🐶 Training

Configure the training environment from configs/examples/UniMRG and set GPU from train.sh.

Start the training process:

bash train.sh

Checkpoints are saved in:

work_dirs/UniMRG/
├── iter_500.pth
├── iter_1000.pth
└── ...

🚀 Evaluation

Visualization

Use the script below to visualize outputs from the Harmon model. Set --mode to pixel, depth, or segmentation.

# Base model checkpoint — predict depth
python scripts/visualization.py \
    --input_image 'assets/image.jpg' \
    --checkpoint 'checkpoints/harmon_1.5b.pth' \
    --mode depth

# Fine-tuned checkpoint — predict depth
python scripts/visualization.py \
    --input_image 'assets/image.jpg' \
    --checkpoint 'work_dirs/UniMRG/iter_4000.pth' \
    --mode depth

GenEval

# GenEval evaluation
export PYTHONPATH=.
torchrun \
  --nnodes 1 \
  --node_rank 0 \
  --nproc-per-node 8 \
  --master_addr 127.0.0.1 \
  --master-port 12345 \
  scripts/parallel_geneval.py \
  --checkpoint checkpoints/harmon_1.5b.pth \
  --batch_size 12 \
  --outdir "results/UniMRG_gen" \
  --mode geneval \
  --image_size 512 \
  --validation_prompts_file ../Benchmark/geneval/evaluation_metadata.jsonl

For score computation, see GenEval.

DPGBench

# DPGBench evaluation
export PYTHONPATH=.
torchrun \
  --nnodes 1 \
  --node_rank 0 \
  --nproc-per-node 8 \
  --master_addr 127.0.0.1 \
  --master-port 12345 \
  scripts/parallel_geneval.py \
  --checkpoint checkpoints/harmon_1.5b.pth \
  --batch_size 4 \
  --outdir "results/UniMRG_dpg" \
  --mode dpgbench \
  --image_size 512 \
  --prompts_file ../Benchmark/dpgbench/prompts.json

For score computation, see DPGBench.

Understanding

Understanding benchmarks follow VLMEvalKit. Clone the repository.

git clone https://github.com/open-compass/VLMEvalKit.git
cd VLMEvalKit

The current VLMEvalKit release does not support Harmon. Apply the following changes:

1. Copy vlmeval/vlm/harmon.py from this repo to the same path in VLMEvalKit (defines Harmon model inference).
2. Replace VLMEvalKit's vlmeval/vlm/__init__.py with ours. The only change is adding from .harmon import Harmon.
3. Replace VLMEvalKit's vlmeval/config.py with ours. The only changes are defining harmon_series and registering it in model_groups.

Default harmon_series:

harmon_series = {
    "Harmon": partial(
        Harmon,
        model_path="Harmon/configs/models/qwen2_5_1_5b_kl16_mar_h.py",
        checkpoint_path="Harmon/checkpoints/harmon_1.5b.pth",
    ),
}

You can edit checkpoint_path to evaluate other checkpoints. Then run:

cd VLMEvalKit
python run.py \
  --data MMBench_DEV_EN MMVP HallusionBench RealWorldQA VSR-zeroshot \
  --model Harmon \
  --work-dir ./outputs

🙌🏻 Acknowledgement

Our code is based on these awesome repos:

• RecA
• Harmon

📖 BibTeX

If you find our repo helpful, please consider leaving a star or cite our paper :)

@article{su2026generation,
  title={Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation},
  author={Su, Zihan and Wei, Hongyang and Cen, Kangrui and Wang, Yong and Chen, Guanhua and Yuan, Chun and Chu, Xiangxiang},
  journal={arXiv preprint arXiv:2601.21406},
  year={2026}
}