Benchmark Evaluation

April 21, 2025 ยท View on GitHub

We provide the scripts to evaluate our model on GenEval and DPG-Bench under ./scripts/eval:

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/eval/bench_dpg.sh

Please follow the instructions in their repo to calculate the metrics.

Generate Images with Our Model

We provide a script to generate images with SimpleAR:

python3 generate.py --prompts "Your prompt"

serving with vLLM [Optional]

If you want to use vllm to accelerate image generation, please install it from this repo, we implement classifier-guidance free (CFG) since it is quite important for visual generation:

git clone https://github.com/wdrink/vllm

cd vllm

pip install -e .

cd ..

mv vllm vllm_local

mv vllm_local/vllm ./

# reinstall transformer here

pip install "transformers@git+https://github.com/huggingface/transformers.git@7bbc62474391aff64f63fcc064c975752d1fa4de"

then just pass --vllm_serving to generate.py to try vLLM.

sampling with SJD

We also implement speculative jacobi decoding (SJD), you can try it with --sjd_sampling.