⭐️ News

• [2026/5/1] GenExam is accepted to ICML 2026.
• [2026/4/22] GPT-Image-2 achieves new SOTA of 74.6 strict score and 93.8 relaxed score.
• [2026/2/26] Results of Seedream 5.0 and Nano Banana 2 are updated.
• [2026/1/28] Results of Qwen-Image-2512 and FLUX.2 dev are updated.
• [2025/12/17] Results of GPT-Image-1.5, Seedream 4.5 and FLUX.2 max are updated.
• [2025/11/23] Nano Banana Pro achieves new SOTA! (72.7 strict score and 93.7 relaxed score)
• [2025/10/7] Results of HunyuanImage-3.0 are updated.
• [2025/9/18] GenExam is released!

📖 Introduction

Exams are a fundamental test of expert-level intelligence and require integrated understanding, reasoning, and generation. Existing exam-style benchmarks mainly focus on understanding and reasoning tasks, and current generation benchmarks emphasize the illustration of world knowledge and visual concepts, neglecting the evaluation of rigorous drawing exams.

We introduce GenExam, the first benchmark for multidisciplinary text-to-image exams, featuring 1,000 samples across 10 subjects with exam-style prompts organized under a four-level taxonomy. Each problem is equipped with ground-truth images and fine-grained scoring points to enable a precise evaluation of semantic correctness and visual plausibility.

Experiments on 17 text-to-image and unified models demonstrate the great challenge of GenExam and the huge gap where open-source models consistently lag behind the leading closed-source ones. By framing image generation as an exam, GenExam offers a rigorous assessment of models' ability to integrate understanding, reasoning, and generation, providing insights on the path to intelligent generative models. Our benchmark and evaluation code will be released.

🚀 Leaderboard

Strict Score

Relaxed Score

Comparison Across Four Dimensions

🛠️ Usage

Our data is stored in data/. You can also download them from Huggingface. Additionally, images organized by taxonomy can be found here.

1. Prerequisites

1. Install requirements: pip install requests tqdm pillow

2. Set openai_api_key and openai_base_url (optional, if you want to use proxy) in run_eval.py for the gpt-5-20250807 evaluator and inference of gpt-image-1.

3. Generate the images offline with your model based on the prompt values in data/annotations/All_Subjects.jsonl. Saved image paths should be like gen_imgs/{id}.png.

2. Run Evaluation

Offline Inference

Run evaluation offline if images are already generated in gen_imgs/:

python run_eval.py --data_dir ./data/ --img_save_dir ./gen_imgs --eval_save_dir ./eval_results

The eval results are saved to separate jsons under ./eval_results for each sample.

The run_eval.py script supports resuming from breakpoints. If your evaluation encounters an error midway, simply re-run the script.

Online Inference

Alternatively, you can add --run_inference to inference and evaluation together (generate images online):

python run_eval.py --run_inference --data_dir ./data/ --img_save_dir ./gen_imgs --eval_save_dir ./eval_results

This script runs gpt-image-1 by default, which costs $185 on the full set ($160 for inference and $25 for evaluation). You can replace the inference_function in the script with customized function for your model's inference.

Speed Up with Multiprocessing

Add a --max_worker argument to speed up with multiprocessing:

python run_eval.py --max_worker 20 --data_dir ./data/ --img_save_dir ./gen_imgs --eval_save_dir ./eval_results

3. Calculate Scores

Run the script to generate a detailed report for the eval results:

python cal_score.py --eval_results_dir ./eval_results

This should give a report like:

================================================================================
Each score dimension:
- semantic_correctness: 0.47
- spelling: 1.48
- readability: 1.55
- logical_consistency: 0.7
================================================================================
Each score dimension (average) for each subject:
- Computer_Science:
  semantic_correctness: 0.53
  spelling: 1.68
  readability: 1.43
  logical_consistency: 0.66
- Physics:
  semantic_correctness: 0.4
  spelling: 1.7
  readability: 1.41
  logical_consistency: 0.5
- Biology:
  semantic_correctness: 0.72
  spelling: 1.28
  readability: 1.59
  logical_consistency: 1.02
- History:
  semantic_correctness: 0.53
  spelling: 1.32
  readability: 1.68
  logical_consistency: 0.85
- Math:
  semantic_correctness: 0.24
  spelling: 1.5
  readability: 1.65
  logical_consistency: 0.29
- Geography:
  semantic_correctness: 0.62
  spelling: 1.27
  readability: 1.69
  logical_consistency: 0.98
- Economics:
  semantic_correctness: 0.56
  spelling: 1.77
  readability: 1.58
  logical_consistency: 0.75
- Chemistry:
  semantic_correctness: 0.33
  spelling: 1.33
  readability: 1.52
  logical_consistency: 0.6
- Music:
  semantic_correctness: 0.26
  spelling: 1.42
  readability: 1.5
  logical_consistency: 0.46
- Engineering:
  semantic_correctness: 0.56
  spelling: 1.49
  readability: 1.43
  logical_consistency: 0.94
--------------------------------------------------------------------------------
Total number of eval results:  487
--------------------------------------------------------------------------------
Strict score:
- Computer_Science(47 samples): 10.2% - Physics(46 samples): 3.5% - Biology(46 samples): 12.2% - History(41 samples): 5.9% - Math(52 samples): 0.0% - Geography(52 samples): 7.7% - Economics(52 samples): 3.1% - Chemistry(52 samples): 4.6% - Music(52 samples): 0.0% - Engineering(47 samples): 6.8% 
Average strict score: 5.4%
--------------------------------------------------------------------------------
Relaxed score:
- Computer_Science(47 samples): 44.8% - Physics(46 samples): 36.9% - Biology(46 samples): 56.1% - History(41 samples): 45.4% - Math(52 samples): 27.2% - Geography(52 samples): 50.7% - Economics(52 samples): 47.6% - Chemistry(52 samples): 32.4% - Music(52 samples): 27.8% - Engineering(47 samples): 47.0% 
Average relaxed score: 41.6%

Run on GenExam-Mini

To run evaluation on the mini subset, you can add a --mini argument when running run_eval.py:

python run_eval.py --mini --data_dir ./data/ --img_save_dir ./gen_imgs --eval_save_dir ./eval_results

If you have already run evaluation on the full set, you can alternatively add --mini when running cal_score.py:

python cal_score.py --mini --eval_results_dir ./eval_results

🖼 Examples of Generated Images

For more examples, please refer to the appendix in our paper and this repo.

Images Generated by Nano Banana Pro

📃 License

This project is released under the MIT license.

🖊️ Citation

If you find our work helpful, please consider giving us a ⭐ and citing our paper:

@article{GenExam,
  title={GenExam: A Multidisciplinary Text-to-Image Exam},
  author = {Wang, Zhaokai and Yin, Penghao and Zhao, Xiangyu and Tian, Changyao and Qiao, Yu and Wang, Wenhai and Dai, Jifeng and Luo, Gen},
  journal={arXiv preprint arXiv:2509.14232},
  year={2025}
}