πŸš€ MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation

July 16, 2025 Β· View on GitHub

arXiv Project Page Hugging Face

πŸ† Efficient Autoregressive Multimodal Image Generation with 10Γ— Less Data

MENTOR Overview

πŸ“„ Paper | πŸ’» Code | πŸ€— Models | πŸ€— Stage1 Data| | πŸ€— Stage2 Data|

🎯 Why MENTOR?

πŸ”₯ 10Γ— Less Training Data ⚑ Faster Training πŸ’ͺ Better Performance
3M vs 16-200M samples 1.5 days in 8*A100 GPU 0.47 CPΒ·PF score

MENTOR demonstrates competitive multimodal image generation capabilities, achieving superior results with dramatically reduced resources thanks to an efficient tuning paradigm. While competitors like Emu2 require 37 billion parameters and vast datasets, MENTOR surpasses their performance with only 2.3 billion parameters and significantly less training data in an autoregressive vision generation framework.

πŸ“‹ Table of Contents

✨ Key Features

FeatureMENTORDiffusion-Based Models
Training Efficiencyβœ… 1.5 days on 8 GPUs❌ 3+ days on 256 GPUs
Deterministic Controlβœ… Precise AR generation❌ Stochastic sampling
Modality Balanceβœ… Lowest CP/PF ratio (0.65)❌ High imbalance (>1.0)
Architectureβœ… Simple unified transformer❌ Complex auxiliary modules

πŸš€ Quick Start

πŸ’» Installation

Requirements
  • Python β‰₯ 3.11
  • PyTorch β‰₯ 2.5.1
  • CUDA β‰₯ 12.1
  • 8Γ— NVIDIA A100 GPUs (80GB) for training
# Clone repository
git clone https://github.com/HaozheZhao/MENTOR.git
cd MENTOR

# Install dependencies
conda env create --file environment.yml

πŸ“₯ Download Dataset and Model

# Download model checkpoints
huggingface-cli download BleachNick/Mentor --local-dir Mentor

# Download Stage-1 dataset
huggingface-cli download BleachNick/Mentor_Stage1 --repo-type dataset --local-dir Mentor_Stage1
cd Mentor_Stage1
cat stage1_data.tar.gz.part-* | pv | tar -xzf -
cd ..

# Download Stage-2 dataset
huggingface-cli download BleachNick/Mentor_Stage2 --repo-type dataset --local-dir Mentor_Stage2
cd Mentor_Stage2
cat stage2_data.tar.gz.part-* | pv | tar -xzf -
cd ..

πŸ‹οΈβ€β™‚οΈ Training

# Stage 1 training
bash scripts/autoregressive/train_stage1.sh

# Stage 2 training
bash scripts/autoregressive/train_stage2.sh

# Run ablation experiments
bash scripts/autoregressive/ablation.sh

πŸ“Š Main Results

πŸ… DreamBench++ Benchmark Leadership

Performance Comparison

MethodModel SizeTraining DataCP↑PF↑CPΒ·PF↑CP/PF↓
DreamEngine10.5B21M0.680.370.261.84
Kosmos-G3B200M0.540.510.281.06
Emu237B16M0.530.690.360.77
IP-Adapter ViT-G2.5B10M0.590.640.380.92
MENTOR2.3B3M0.550.840.470.65

CP: Concept Preservation | PF: Prompt Following | Lower CP/PF = Better Balance

🎨 Superior Image Reconstruction

MethodCOCO L2↓JourneyDB L2↓Improvement
SeedTokenizer0.51020.5291\
SEED-X0.43170.4352\
EMU2-Gen0.38280.2869\
DreamEngine0.20650.2052Baseline
MENTOR0.10080.0867~50% Better

πŸ—οΈ Method Overview

Two-Stage Training Paradigm

Model Architecture

StagePurposeTasksTraining Time
Stage 1Multimodal Alignmentβ€’ Image Reconstruction
β€’ Object Segmentation
β€’ T2I Generation		14 hours
Stage 2Instruction Tuningβ€’ Image Recovery
β€’ Subject-driven Gen
β€’ Balanced Integration		20 hours

Architecture Components

πŸ”§ Click to expand technical architecture

Multimodal Encoder

  • Vision: CLIP-Large-Patch14 (frozen)
  • Language: FlanT5-XL
  • Connector Options:
    • MLP-based: Full visual detail preservation (256 tokens/image)
    • Query-based: Efficient token compression (32 tokens/image)

Autoregressive Decoder

  • Base Model: LlamaGen-XL (775M parameters)
  • Vocabulary: Shared with VQGAN tokenizer
  • Generation: Deterministic next-token prediction

🎯 Usage Examples

Basic Generation

  python demo.py \
      --image_path cat.jpg \
      --prompt "A cat in <image>.\n A cat in a 16-bit fantasy pixel-art scene" \
      --gpt_ckpt Mentor/stage2.pt \
      --vq_ckpt  Mentor/vq_ds16_t2i.pt \
      --output out/cat_pixel.jpg \
      --mm_vision_tower "openai/clip-vit-large-patch14" \
      --multimodal_encoder llava \

πŸ”¬ Technical Details

πŸ“ Mathematical Formulation

Training Objective

Given multimodal inputs c = {I, T}, the encoder Ο† produces:

H = MLP(Ο†(c)) = (h₁, ..., hβ‚˜) ∈ β„α΄ΉΛ£α΅ˆ

The AR decoder ΞΈ generates image sequence y:

ΞΈ(y | H) = βˆα΅’β‚Œβ‚α΄Έ ΞΈ(yα΅’ | y<α΅’, H)

Classifier-Free Guidance

  • Training: Replace H with Hα΅€ with probability p = 0.1
  • Inference: β„“_g = β„“_u + (β„“_c - β„“_u) Γ— Ξ»
πŸ”§ Training Configuration

Hyperparameters

  • Optimizer: AdamW
  • Learning Rate: Stage 1: 5e-4, Stage 2: 1e-4
  • Batch Size: 96 for stage1 and 64 for stage 2(global)
  • Warmup: 5% of total steps
  • Schedule: Cosine decay

Hardware Requirements

  • Training: 8Γ— A100 80GB (~34 hours total)
  • Inference: 1Γ— GPU 24GB+
πŸ“Š Extended Results

Ablation Studies

ConfigurationCPPFCPΒ·PF
Full Model0.5550.8390.466
w/o Stage 10.1790.6730.120
w/o Image Recovery0.6610.2840.188
w/o Segmentation0.4120.9180.378

πŸ› οΈ ROPE Fix for LlamaGen #54

We identified and resolved a critical bug in LlamaGen's 2D ROPE implementation that was causing significant information loss. Our fix required retraining the model prior to fine-tuning.

View more examples

Image Reconstruction

Image Reconstruction Examples

Multi-Subject Generation

Multi-Subject Generation

In-Context Learning

In-Context Learning Examples

πŸ“š Citation

If you find MENTOR useful, please cite our paper:

@misc{zhao2025mentorefficientmultimodalconditionedtuning,
      title={MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Models}, 
      author={Haozhe Zhao and Zefan Cai and Shuzheng Si and Liang Chen and Jiuxiang Gu and Wen Xiao and Junjie Hu},
      year={2025},
      eprint={2507.09574},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2507.09574}, 
}

πŸ™ Acknowledgments

We thank the teams behind [LlamaGen]{https://github.com/FoundationVision/LlamaGen}, and [DreamBench++]{https://github.com/yuangpeng/dreambench_plus} for their foundational contributions.