image-bench-ai

August 26, 2026 · View on GitHub

Website Stars

Reproducibility artifacts for ImageBench — a public leaderboard for text-to-image models. This repo holds the frozen prompt suites and judge questions used to score models; the results, the generated images, and the interactive leaderboard live on the site.

Latest results: imagebench.ai — 50+ models scored across every category, every generated image published in the gallery, nothing cherry-picked.

What ImageBench is

  • A fixed benchmark: same 192 generation prompts, same specialized judge questions, same scoring protocol for every model.
  • A capability probe, not a beauty contest — the categories target things text-to-image models actually get wrong (text rendering, counting, negation, hands, physics, reflections, world knowledge).
  • Automated grading by vision-language model (VLM) judges. Every image gets a binary PASS/FAIL verdict on a concrete, test-specific question.
  • Everything is published: every generated image, every judge verdict, every aggregated score. No selection, no retries, no post-hoc tuning per model.

The headline Overall score is a blend:

Overall = 0.5 × Capability + 0.5 × EPS

where Capability is the VLM PASS rate over the 192 tests and EPS (Estimated Preference Score) is an aesthetic axis from the HPSv3 human-preference reward model. Details: imagebench.ai/blog/benchmark-v1-methodology.

The versions

ImageBench is a single benchmark that has evolved. All versions share the same 64 tests and 192 generation prompts — what changes is how the judge grades them.

VersionJudgeWhat changedRead more
V1Multi-VLM router (Qwen3-VL, Qwen3.5-122B, Gemma4-26B, Qwen3.6-27B) per categoryOriginal release. Generic per-category question template with a {prompt_excerpt} placeholder — the judge sees the generation prompt and interprets it.imagebench-v1/ · methodology
V1.1Gemini 3.1 Pro (single frontier judge)Specialized, standalone question per prompt variant (question_1/2/3). The judge never sees the generation prompt — the question bakes in the ground truth. Stricter, more consistent grading; scores drop ~22 points but the ranking largely holds.imagebench-v1.1/ · blog post
V1.2V1.1 + Qwen3.5-122B on HandsGemini over-flagged hands (F1 65% vs Qwen3.5-122B's 80% on human-labeled hands). V1.2 routes only the 12 Hands tests to Qwen3.5-122B; the other 180 tests stay with Gemini. Human-realism scores rise ~18 points; ranking barely moves.Same prompts as V1.1 · blog post

V1.2 does not add a new prompt file — it reuses imagebench-v1.1/prompts.csv and only changes the per-category judge routing. Prompts and judge questions are frozen across versions; only the grader varies.

Repo layout

imagebench-v1/
  challenges.csv            # V1 frozen challenge suite (192 rows, one per prompt variant)

imagebench-v1.1/
  prompts.csv               # V1.1 benchmark definition (64 tests × 3 variants, one row per test)
  categories.csv            # category / subcategory / difficulty descriptions
  METHODOLOGY.md            # V1.1 spec, schema, and how a judge sees a prompt

imagebench-v1/challenges.csv and imagebench-v1.1/prompts.csv are two shapes of the same prompts:

  • V1 challenges.csv is flattened — one row per (test × variant), so 192 rows. Each row carries the resolved vision_question for that specific variant using the V1 generic template, plus the V1 router's assigned VLM.
  • V1.1 prompts.csv is wide — one row per test (64 rows), with prompt_1/2/3 and the matching specialized question_1/2/3 side by side. The vision_question_template column keeps the V1 generic form as a fallback for any variant whose specialized question is blank.

If you're reproducing V1 exactly, use the V1 file. If you're running the current benchmark or adding a new specialized question, use the V1.1 file.

Reproducing a run

  1. Generate 192 images from either imagebench-v1/challenges.csv (one image per row) or imagebench-v1.1/prompts.csv (three images per row — one per prompt_N).
  2. Ask a VLM judge the corresponding vision_question (V1) or question_N (V1.1) about each image, with the system-prompt convention that the judge ends its response with VERDICT: PASS or VERDICT: FAIL.
  3. Aggregate: score = PASS count / images evaluated, reported overall and per category / subcategory / difficulty.
  4. Compare against the published runs at imagebench.ai. Every image and every verdict for the 50+ scored models is downloadable via the gallery pages.

For the aesthetic EPS axis, run the images through HPSv3 and blend as Overall = 0.5 × Capability + 0.5 × EPS.

Attribution

If you use these prompts or reproduce ImageBench results, please cite imagebench.ai and link back.