MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation

August 25, 2026 · View on GitHub

Can multimodal generative models reason through the images and videos they produce — or do they mainly render plausible-looking output?

What is MMGR
Why another benchmark
Benchmark at a glance
Headline results
Repository structure
Installation
Credentials
Getting the data
The three-stage pipeline
Reproducing each task
Evaluation protocol
How reliable is each number
Evaluating your own model
Cost and compute
Known limitations
FAQ
Citation
Acknowledgements

What is MMGR

Modern text-to-image and text-to-video systems render photorealistic content directly from natural language. But visual fluency is not reasoning. A generated video can look smooth while violating causal order, object permanence, or task constraints; a generated image can look plausible while failing the logical or spatial relation the prompt required.

MMGR evaluates whether a generative model can solve a reasoning problem by generating its solution. It covers 10 tasks across 3 domains, totalling 1,853 task instances, and probes 5 reasoning abilities: Physical, Logical, 2D Spatial, 3D Spatial, and Temporal.

Two properties distinguish it:

Answer-verifiable scoring wherever reliable ground truth exists. Maze is scored by pixel-level path reconstruction; Sudoku by OCR followed by exact-match and constraint checks. No judge model is in the loop for these.
Process-aware, chain-of-frame evaluation for video. Intermediate frames must be valid steps toward the target outcome, not merely smooth transitions between plausible visual states. This is what lets MMGR ask whether temporal generation provides a real reasoning advantage — or instead introduces new failures from maintaining cross-frame consistency.

Tasks are run against video generators, image generators, and text-only LLM/VLM baselines wherever each is applicable, which is what makes it possible to separate reasoning failures from modality-specific rendering failures. Coverage is not uniform across the three classes: video models are evaluated on all 10 tasks, image models on the 8 non-physics tasks, and the Gemini text-only baselines on only 3 — Maze, Sudoku and Math.

Why another benchmark

Benchmark family	Mod.	Process	Rule-ver.	V+I+L	Embod.	Primary scope
WISE / Commonsense-T2I	I	✗	✗	✗	✗	Knowledge, physical, spatial T2I
RISEBench / KRIS-Bench	I	✗	✗	✗	✗	Reasoning-informed image editing
GIR-Bench / RBench-V	I	✗	~	✗	✗	Unified reason-and-generate
VBench / EvalCrafter	V	✗	✗	✗	✗	Quality, alignment, human preference
T2V-CompBench / VideoPhy	V	✗	~	✗	✗	Compositional, physical, world-model fidelity
VIPER	V	✓	✗	✗	✗	Process–outcome consistency
VBVR	V	✓	✓	✗	✗	Rule-scored synthetic video reasoning
MMGR (ours)	V+I+L	✓	✓†	✓	✓	Abstract, embodied, physical; cross-modal

Mod.: Video / Image / LLM-VLM generators. Process: scores intermediate steps, not only the final clip/image. Rule-ver.: deterministic rule-based scoring vs. only human/VLM judgment. V+I+L: covers all three generator types in one framework. Embod.: real-world agent-centric navigation. ✓ yes, ✗ no, ~ partial; † deterministic checkers where reliable rules/geometry exist, VLM judges otherwise.

Benchmark at a glance

Task	Phy.	Log.	3D Spa.	2D Spa.	Temp.	# Samples
D1: Abstract Reasoning						1,323
T1 · Maze		✓		✓	✓	240
T2 · Sudoku		✓		✓		300
T3 · ARC		✓		✓	✓	456
T4 · Math		✓				327
D2: Embodied Navigation						480
T5 · Panoramic View Last-Mile Nav.	✓		✓		✓	120
T6 · Top-down View Real-World Nav.	✓			✓	✓	120
T7 · 3D Real-World Nav.	✓		✓		✓	120
T8 · SLAG	✓		✓	✓	✓	120
D3: Physical Commonsense						50
T9 · Physical Concepts	✓		✓		✓	25
T10 · Sports	✓		✓		✓	25
Total						1,853

Every task ships with granular difficulty control — maze grid size and generator algorithm, Sudoku grid size and clue sparsity, ARC version and shape-consistency, math source and tier, navigation floor complexity / view fidelity / path length / goal specification. See docs/TASKS.md for the full specification of every difficulty axis and its bucket sizes.

Headline results

Zero-shot performance under each task's gated primary metric (%), aggregated over all difficulty levels. Maze and Sudoku use deterministic pixel- and OCR-based evaluators; the rest use the rubric-based VLM evaluator. Gemini columns are text-only solver baselines, not multimodal generation systems, and are therefore excluded from the bold — bold marks the best generative model per task. On Maze, Sudoku and Math the text-only baselines score higher than every generative model; that gap is the point, not an oversight.

Task	Veo-3	Sora-2	Wan-2.2	Nano-banana	Nano-banana Pro	GPT-4o-image	GPT-image-1.5	Qwen-image	Gemini-3-Flash	Gemini-3-Pro
D1: Abstract Reasoning
Maze	3.67	0.00	0.00	3.67	9.17	0.00	0.97	0.14	55.67	59.33
Sudoku	0.00	0.00	0.00	8.60	7.00	1.33	6.78	0.00	65.53	63.75
ARC	4.80	11.67	0.15	8.15	28.07	0.00	12.72	2.15	–	–
Math	10.61	10.56	0.00	11.47	72.69	26.24	25.52	11.22	71.38	74.14
D2: Embodied Navigation
Last-Mile Nav.	60.00	0.00	14.17	74.17	75.83	0.00	55.84	16.67	–	–
Top-down View Nav.	19.49	3.39	5.09	11.11	33.05	3.39	26.27	5.08	–	–
3D R.-W. Nav.	22.50	0.00	24.17	79.17	85.00	13.33	77.92	38.33	–	–
SLAG	11.02	12.50	0.85	28.79	37.29	16.67	31.36	6.78	–	–
D3: Physical Commonsense
Physical Concepts	41.67	76.00	26.67	–	–	–	–	–	–	–
Sports	60.00	64.00	21.33	–	–	–	–	–	–	–

"–" marks a model that was not evaluated or for which no usable result file was available. Full fine-grained tables (per difficulty level, per subset, per failure mode) are in docs/RESULTS.md.

Three findings worth stating plainly:

Symbolic reasoning is near-floor for video models. Every video model scores 0.00% on Sudoku, below 12% on ARC, below 11% on Math. Visually coherent generation does not preserve the state, rules, and intermediate deductions that abstract reasoning requires.
Longer visual generation ≠ stronger reasoning. Image generators win all four navigation tasks — Nano-banana Pro leads Last-Mile (75.83), Top-down (33.05), 3D R.-W. (85.00), and SLAG (37.29) — despite producing no temporal output at all. Video models are competitive only in short-horizon local navigation (Veo-3 at 60.00 on Last-Mile) and degrade sharply once the viewpoint or horizon changes.
Video models are strongest where the task resembles natural video. Sora-2 leads Physical Concepts (76.00) and Sports (64.00). But this domain has no deterministic checker, so read these as judged physical plausibility, not verified physical correctness.

The single diagnosis across all three: current generators are better at local visual plausibility than at maintaining a persistent task state.

Repository structure

MMGR/
├── README.md                     ← you are here
├── requirements.txt
├── dataset_manifest.json         ← machine-readable release index (copy of the dataset manifest)
├── docs/
│   ├── TASKS.md                  ← full spec of all 10 tasks + difficulty axes
│   ├── METRICS.md                ← every fine-grained metric, and how the gated score is formed
│   └── RESULTS.md                ← main table + all fine-grained appendix tables
└── mmgr/
    ├── common/
    │   └── sora2/                ← Azure Sora-2 API client
    │        sora_client.py            reusable client class
    │        job_monitor.py            create / check / monitor / list one job
    │        batch_generate_parallel.py  batch generation with checkpoint + retry
    │        download_all_videos.py    fetch every completed job's output
    ├── abstract_reasoning/
    │   ├── maze/
    │   │   ├── data_generation/          maze synthesis (DFS + Wilson), solution & metadata
    │   │   ├── generation/               Veo-3 / Nano-banana runners + prompt templates
    │   │   ├── pixel_based_evaluation/   ← deterministic scorer used for the headline number
    │   │   └── vlm_based_evaluation/     Gemini judge + rubric prompts
    │   ├── sudoku/
    │   │   ├── data_generation/          4×4 / 9×9 puzzle synthesis
    │   │   ├── generation/
    │   │   ├── ocr_based_evaluation/     ← deterministic scorer used for the headline number
    │   │   └── vlm_based_evaluation/
    │   ├── arc/
    │   │   ├── generation/
    │   │   └── vlm_based_evaluation/
    │   └── math/
    │       ├── data_generation/          GSM8K / MATH500 / AIME / Omni-MATH → rendered problem images
    │       ├── generation/
    │       └── vlm_based_evaluation/
    ├── embodied_navigation/
    │   ├── data_generation/              HM3D download, region captioning, test-table generation
    │   ├── generation/                   ← ONE runner for all four tasks, selected via --task
    │   ├── vlm_based_evaluation/         ← ONE evaluator for all four tasks
    │   ├── panoramic_view_last-mile_navigation/   (T5)
    │   ├── top-down_view_real-world_navigation/   (T6)
    │   ├── 3D_real-world_navigation/              (T7)
    │   └── simultaneous_localization_and_generation/  (T8)
    │        each of the four holds:
    │          prompt_generation/     builds per-scene input prompts
    │          generation_prompts/    the Veo-3 / Nano-banana prompt templates
    │          vlm_based_evaluation/  judge prompt + criteria + output format
    └── physical_commonsense/
        ├── data_generation/          VideoPhy sampling + sports prompt construction
        ├── generation/
        └── vlm_based_evaluation/

A note on the embodied layout. The navigation runners are shared: a single veo_generation.py / nano_banana_generation.py / veo_evaluation.py / nano_banana_evaluation.py handles all four tasks and you pick one with --task. Only the prompts, criteria and per-scene prompt builders are task-specific. The directory tree reflects that real topology rather than duplicating four copies of the same runner.

In the original research code the four navigation tasks are numbered task01–task04. The mapping to the paper's names, verified against the prompt text itself, is:

Code	Paper task	Evidence in the prompt template
`task01`	T5 · Panoramic View Last-Mile Navigation	"last mile" navigation within a 360-degree panoramic scene
`task02`	T6 · Top-down View Real-World Navigation	agent navigating a 2D top-down environment
`task03`	T7 · 3D Real-World Navigation	"cutaway" or "dollhouse" style … third-person isometric view
`task04`	T8 · SLAG	split-screen … LEFT = 3D dollhouse view, RIGHT = generated 2D map

You will still see these numbers in the code. Mind the two spellings — they are not interchangeable:

--task takes task_01 … task_04 (with an underscore). The runner strips the task_ prefix to build the manifest filename, so passing task01 silently looks for tasktask01_test_table.json and fails.
The manifest files themselves are named task01_test_table.json (no underscore).

Installation

git clone https://github.com/Zefan-Cai/MMGR.git
cd MMGR
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Python 3.10+ is recommended.

Two components have heavier, optional dependencies:

Sudoku OCR evaluation needs an OCR backend (--ocr_method paddleocr by default) and benefits from a GPU. mmgr/abstract_reasoning/sudoku/ocr_based_evaluation/video_gen_evaluation.py supports multi-GPU sharding via --gpu_ids and --num_workers.
Math problem rendering shells out to a LaTeX toolchain for the AIME and MATH-500 renderers. Install a TeX distribution (e.g. TeX Live or MacTeX) if you intend to regenerate those images rather than download them.

Credentials

No API key is stored in this repository. Every runner reads credentials from the environment.

Variable	Used by	Notes
`GOOGLE_APPLICATION_CREDENTIALS`	all Veo-3 generation, all Gemini evaluation	Path to a Vertex AI service-account JSON
`GOOGLE_API_KEY`	Nano-banana generation	Accepted as an alternative to the service account
`VERTEX_PROJECT_ID`	all Gemini evaluation	Vertex AI project
`VERTEX_LOCATION_ID`	all Gemini evaluation	Vertex AI region
`AZURE_OPENAI_API_KEY`	`mmgr/common/sora2/*`	Azure Sora-2 deployment key
`OPENAI_API_KEY`	`mmgr/embodied_navigation/data_generation/region_caption.py`, maze prompt enrichment	Only needed for data (re)generation
`AZURE_SORA_BASE_URL`	`sora2/job_monitor.py`	Full jobs URL, e.g. `https://HOST/openai/v1/video/generations/jobs`
`AZURE_OPENAI_BASE_URL`	`sora2/batch_generate_parallel.py`	Full base URL, e.g. `https://HOST/openai/v1`
`AZURE_OPENAI_ENDPOINT`	`sora2/download_all_videos.py`	Host only, e.g. `https://HOST`
`AZURE_SORA_ENDPOINT`	`sora2/sora_client.py`	Full jobs URL

The three Sora entry points each read a different variable and two of them want a full API path, not just the host. Export all of them; all ship as YOUR-RESOURCE placeholders.

Paths are configurable through the environment too, so nothing is pinned to one machine:

Variable	Meaning	Default
`VR_DATA_ROOT`	Root for benchmark data and outputs	`./data`
`SPORTS_DATA_DIR`	Sports prompt input/output directory	`./data/sports`
`MM_OUTPUT_DIR`	Output root for the maze multimodal generator	`./outputs`
`LATEX_BIN_PATH`	Directory holding `pdflatex`, prepended to `PATH`	empty (use system `PATH`)
`MM_DEFAULT_CREDENTIALS_PATH`	Optional GCP credential fallback	empty (disabled)

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
export VERTEX_PROJECT_ID=your-project
export VERTEX_LOCATION_ID=us-central1

Most runners also accept --credentials /path/to/service-account.json to override the environment for a single invocation.

Getting the data

The benchmark inputs (problem images, prompts, ground-truth solutions) are released separately from the code, because they total several gigabytes:

# requires: pip install huggingface_hub
huggingface-cli download ZefanCai/MMGR --repo-type dataset --local-dir ./data

dataset_manifest.json in this repo is the authoritative index. Its per-task inventory:

Task	Domain	Input images	Solution files	Metadata
arc	abstract_reasoning	456	456	not provided
math	abstract_reasoning	331	7 †	not provided
maze	abstract_reasoning	720 ‡	720	not provided
sudoku	abstract_reasoning	300	300	not provided
panoramic_view_last-mile_navigation	embodied_navigation	120	0 §	partial
top-down_view_real-world_navigation	embodied_navigation	120	0 §	partial
3D_real-world_navigation	embodied_navigation	120	0 §	partial
simultaneous_localization_and_generation	embodied_navigation	120	0 §	partial
physical_concept	physical_commonsense	0 ¶	0 ¶	in prompt JSON
sports	physical_commonsense	0 ¶	0 ¶	in prompt JSON

† Math solutions are aggregated JSON files, not one file per problem — 7 files covering all 327 problems, holding both the problem statement and the reference solution.

‡ The maze image count exceeds the 240 evaluation instances because each instance ships multiple rendered artifacts (problem image and solution image variants).

§ Navigation ground-truth solutions are not in the public mirror. The source ground-truth mapping still requires approval before publication. You can run generation and VLM-based evaluation without them; you cannot run a ground-truth-diffing analysis.

¶ Physical Commonsense is prompt-driven video generation — it intentionally has no input images and no solution images. The prompts live in data/prompt/physical_commonsense/.

Pairing rules (also encoded in the manifest):

ARC, Maze, Sudoku: one input image per ground-truth solution image, paired by filename or by relative directory structure.
Math: rendered problem images plus aggregated solution JSON under data/solution/abstract_reasoning/math/.
Embodied, color-goal samples: the visible target is encoded in the input image and in the matching floor/quality/turn subset directory.
Embodied, object-goal samples: prompts contain a {LOCATION_DESCRIPTION} placeholder. Resolve it from data/prompt/embodied_nav/location_descriptions_manifest.jsonl — 60 object-goal records per navigation task, 240 total.
Physical Commonsense: sports_prompts.json carries prompt, physics_focus, and expected_motion.

The three-stage pipeline

Every task in MMGR follows the same shape:

  ┌──────────────────┐     ┌──────────────────┐     ┌──────────────────────┐
  │ 1. DATA          │ ──▶ │ 2. GENERATION    │ ──▶ │ 3. EVALUATION        │
  │    generation    │     │                  │     │                      │
  │ synthesize or    │     │ Veo-3 / Sora-2 / │     │ deterministic checker│
  │ render problem   │     │ Wan-2.2 /        │     │   (maze, sudoku)     │
  │ images + GT      │     │ Nano-banana /    │     │        or            │
  │                  │     │ GPT-image / Qwen │     │ Gemini-2.5-Pro judge │
  └──────────────────┘     └──────────────────┘     └──────────────────────┘
       data_generation/         generation/          *_based_evaluation/

Stage 1 is optional — download the released data instead and start at stage 2. Stage 3 always emits per-sample JSON with the fine-grained metrics plus the gated primary score.

Throughout, five samples are drawn per prompt per model, using default settings for closed models and recommended settings for open ones, with no task-specific tuning.

Reproducing each task

All commands below are run from the repository root. Every flag shown is a real flag of the underlying script — run any script with --help to see its full signature. Paths are written explicitly rather than relying on defaults, so the commands work regardless of where you put the data.

D1 · Abstract Reasoning

T1 · Maze

240 instances: DFS and Wilson mazes, three difficulty levels, four start–goal configurations. A successful solution must not alter the maze, cross no walls, and reach the target. Action reflection is scored as a fine-grained diagnostic metric but is not part of the primary gate — the pixel evaluator instead penalises excessive reflection. (The paper's Table 3 overview lists action reflection among the criteria; the appendix aggregation formula excludes it. We follow the aggregation formula, which is what the reported numbers were computed with.)

(1) Regenerate the data (optional — released data is on HuggingFace)

python mmgr/abstract_reasoning/maze/data_generation/maze_data_generation.py

Writes problem PNGs, solution PNGs, path NPYs and metadata JSON. This script has no CLI flags; edit the configuration block at the top to change grid sizes or counts.

(2) Generate solutions

# Video — Veo-3
python mmgr/abstract_reasoning/maze/generation/veo_generation.py \
    --maze-dir   ./data/image/abstract_reasoning/maze \
    --output-dir ./outputs/maze/veo3 \
    --generator  dfs \
    --level      easy \
    --verbose

# Image — Nano-banana
python mmgr/abstract_reasoning/maze/generation/nano_banana_generation.py \
    --maze-dir   ./data/image/abstract_reasoning/maze \
    --output-dir ./outputs/maze/nano_banana \
    --generator  dfs \
    --level      easy

--generator selects the maze algorithm subset (DFS / Wilson) and --level the difficulty tier.

(3) Evaluate — deterministic, this is the headline number

# Video: extract frames, track the green marker, accumulate the traced path, compare to the solution
python mmgr/abstract_reasoning/maze/pixel_based_evaluation/video_gen_evaluation.py \
    --input_dir    ./outputs/maze/veo3 \
    --solution_dir ./data/solution/abstract_reasoning/maze \
    --output_dir   ./eval/maze/veo3 \
    --model_name   veo3 \
    --num_frames   192 \
    --extraction_method green_marker_tracking

# Image: same four checks applied to the single output frame
python mmgr/abstract_reasoning/maze/pixel_based_evaluation/image_gen_evaluation.py \
    --input_dir    ./outputs/maze/nano_banana \
    --solution_dir ./data/solution/abstract_reasoning/maze \
    --output_dir   ./eval/maze/nano_banana \
    --model_name   nano_banana_pro

The pixel evaluator checks four things with no model in the loop: wall crossing, layout matching (maze structure and start/goal positions preserved), path matching (dilate + IoU against the reference path), and reflection (backtracking / exploration behaviour).

(4) Evaluate — VLM judge (diagnostic; not the headline number)

python mmgr/abstract_reasoning/maze/vlm_based_evaluation/maze_evaluation.py \
    --input_dir    ./outputs/maze/veo3 \
    --solution_dir ./data/solution/abstract_reasoning/maze \
    --output_dir   ./eval/maze/veo3_vlm

T2 · Sudoku

300 puzzles across 4×4 and 9×9 grids, difficulty controlled by clue sparsity. A successful solution must keep the given clues unchanged, violate no constraints, and complete the grid accurately — the primary gate is exactly clues_changed = 0 ∧ constraint_violation = 0 ∧ completion_accuracy = 1. Action reflection is reported as a diagnostic metric and is not gated on.

(1) Regenerate the data (optional)

python mmgr/abstract_reasoning/sudoku/data_generation/sudoku_data_generation.py

(2) Generate solutions

python mmgr/abstract_reasoning/sudoku/generation/veo_generation.py \
    --sudoku-dir ./data/image/abstract_reasoning/sudoku \
    --output-dir ./outputs/sudoku/veo3 \
    --size 9x9 --level medium

python mmgr/abstract_reasoning/sudoku/generation/nano_banana_generation.py \
    --sudoku-dir ./data/image/abstract_reasoning/sudoku \
    --output-dir ./outputs/sudoku/nano_banana \
    --size 9x9 --level medium

(3) Evaluate — deterministic OCR, this is the headline number

# Video (supports multi-GPU sharding)
python mmgr/abstract_reasoning/sudoku/ocr_based_evaluation/video_gen_evaluation.py \
    --video_results_dir ./outputs/sudoku/veo3 \
    --puzzle_dir        ./data/image/abstract_reasoning/sudoku \
    --solution_dir      ./data/solution/abstract_reasoning/sudoku \
    --output_dir        ./eval/sudoku/veo3 \
    --ocr_method  paddleocr \
    --frame_sampling 8 \
    --num_workers 4 --gpu_ids 0,1,2,3

# Image
python mmgr/abstract_reasoning/sudoku/ocr_based_evaluation/image_gen_evaluation.py \
    --image_results_dir ./outputs/sudoku/nano_banana \
    --puzzle_dir        ./data/image/abstract_reasoning/sudoku \
    --solution_dir      ./data/solution/abstract_reasoning/sudoku \
    --output_dir        ./eval/sudoku/nano_banana \
    --ocr_method paddleocr

The OCR evaluator reads digits out of the generated grid, then applies exact match against the clues (were any given digits altered?) and row / column / box constraint checks. For video it runs frame by frame at the sampling rate set by --frame_sampling.

(4) Evaluate — VLM judge (diagnostic)

python mmgr/abstract_reasoning/sudoku/vlm_based_evaluation/veo_evaluation.py \
    --input_dir    ./outputs/sudoku/veo3 \
    --puzzle_dir   ./data/image/abstract_reasoning/sudoku \
    --solution_dir ./data/solution/abstract_reasoning/sudoku \
    --output_dir   ./eval/sudoku/veo3_vlm

T3 · ARC

456 tasks from ARC-AGI v1 and v2, split by shape consistency (match / mismatch) and difficulty. A successful solution must show correct pattern recognition, maintain grid integrity, and apply accurate colors, exactly matching the ground-truth output.

(2) Generate solutions

python mmgr/abstract_reasoning/arc/generation/veo_generation.py \
    --dataset v1 --subset match --level easy \
    --base-benchmark-dir ./data/image/abstract_reasoning/arc \
    --output-dir ./outputs/arc/veo3 \
    --max-concurrent 3

python mmgr/abstract_reasoning/arc/generation/nano_banana_generation.py \
    --dataset v1 --subset match --level easy \
    --base-benchmark-dir ./data/image/abstract_reasoning/arc \
    --output-dir ./outputs/arc/nano_banana

--dataset is v1 / v2, --subset is match / mismatch, --level the difficulty tier — all three are required. Convenience wrappers that sweep every combination are in the same directory: run_veo_generation.sh and run_image_generation.sh.

(3) Evaluate

python mmgr/abstract_reasoning/arc/vlm_based_evaluation/veo_evaluation.py \
    --input_dir        ./outputs/arc/veo3 \
    --ground_truth_dir ./data/solution/abstract_reasoning/arc \
    --output_dir       ./eval/arc/veo3

python mmgr/abstract_reasoning/arc/vlm_based_evaluation/nano_banana_evaluation.py \
    --input_dir        ./outputs/arc/nano_banana \
    --ground_truth_dir ./data/solution/abstract_reasoning/arc \
    --output_dir       ./eval/arc/nano_banana \
    --max_concurrent 5

run_all_image_evaluations.sh batches every image model at once.

T4 · Math

327 problems drawn from GSM8K, MATH-500, AIME 2024/2025 and Omni-MATH, rendered as problem images. A successful solution must be correct in both its intermediate steps and its final answer — the primary gate is exactly final_correctness = 1 ∧ intermediate_correctness = 1. Action reflection is reported as a diagnostic metric and is not gated on.

(1) Render problem images (optional)

python mmgr/abstract_reasoning/math/data_generation/gsm8k_to_images.py
python mmgr/abstract_reasoning/math/data_generation/math500_to_images.py     # needs LaTeX
python mmgr/abstract_reasoning/math/data_generation/omni_math_to_images.py
python mmgr/abstract_reasoning/math/data_generation/aime24_to_images.py      # needs LaTeX
python mmgr/abstract_reasoning/math/data_generation/aime25_to_images.py      # needs LaTeX

These pull from HuggingFace and render to PNG plus a metadata JSON. They have no CLI flags — the output directory is set at the top of each file.

(2) Generate solutions

python mmgr/abstract_reasoning/math/generation/veo_generation.py \
    --image-dir  ./data/image/abstract_reasoning/math \
    --dataset    gsm8k \
    --output-dir ./outputs/math/veo3 \
    --workers 3

python mmgr/abstract_reasoning/math/generation/nano_banana_generation.py \
    --image-dir  ./data/image/abstract_reasoning/math \
    --dataset    gsm8k \
    --output-dir ./outputs/math/nano_banana

For Omni-MATH, use the _omni variants, which take --tx (tier) and --type (domain) instead of --dataset; both are required:

python mmgr/abstract_reasoning/math/generation/veo_generation_omni.py \
    --image-dir ./data/image/abstract_reasoning/math/omni \
    --tx T3 --type applied \
    --output-dir ./outputs/math/omni_veo3

(3) Evaluate

python mmgr/abstract_reasoning/math/vlm_based_evaluation/math_evaluation.py \
    --image_results_dir ./outputs/math/nano_banana \
    --video_results_dir ./outputs/math/veo3 \
    --solution_file     ./data/solution/abstract_reasoning/math/solutions.json \
    --output_dir        ./eval/math

python mmgr/abstract_reasoning/math/vlm_based_evaluation/omni_math_evaluation.py \
    --image_results_dir ./outputs/math/omni_nano_banana \
    --video_results_dir ./outputs/math/omni_veo3 \
    --solution_file     ./data/solution/abstract_reasoning/math/omni_solutions.json \
    --output_dir        ./eval/math_omni \
    --max_concurrent 5

Note that the math evaluators take both an image directory and a video directory in one invocation — they score the two modalities side by side against the same solution file.

All four tasks share one generation runner and one evaluation runner; --task picks which. Each task has 120 samples spanning 24 hard-level configurations that vary floor complexity, view fidelity, path length, and goal specification.

A successful episode must jointly satisfy task completeness (success, oracle success, trajectory alignment), physical understanding (object semantics, agent consistency, spatial alignment), and instruction following (destination integrity, scene consistency).

(1) Build the benchmark from HM3D / Matterport3D (optional, heavy)

python mmgr/embodied_navigation/data_generation/download_hm3d_meta.py
python mmgr/embodied_navigation/data_generation/download_hm3d_image.py
python mmgr/embodied_navigation/data_generation/region_caption.py --source task03
python mmgr/embodied_navigation/data_generation/concat_slag_inputs.py       # builds the T8 split-screen inputs
python mmgr/embodied_navigation/data_generation/generate_test_tables.py     # emits taskNN_test_table.json

Then build the per-scene input prompts for a given task:

python mmgr/embodied_navigation/panoramic_view_last-mile_navigation/prompt_generation/veo3_prompt_generation.py \
    --root ./data/image/embodied_navigation/panoramic_view_last-mile_navigation

(2) Generate solutions

# NOTE the underscore: the runner expects task_01 .. task_04, not task01.
# T5 Panoramic last-mile = task_01 | T6 Top-down = task_02
# T7 3D real-world       = task_03 | T8 SLAG     = task_04
python mmgr/embodied_navigation/generation/veo_generation.py \
    --task task_01 \
    --benchmark-dir ./data/image/embodied_navigation \
    --output-dir    ./outputs/embodied/task01_veo3 \
    --max-concurrent 3 \
    --no-update-json

python mmgr/embodied_navigation/generation/nano_banana_generation.py \
    --task task_01 \
    --benchmark-dir ./data/image/embodied_navigation \
    --output-dir    ./outputs/embodied/task01_nano_banana \
    --skip-existing

⚠️ veo_generation.py writes back to the input manifest by default. On success it updates taskNN_test_table.json in place. Pass --no-update-json to keep your benchmark inputs pristine — recommended unless you deliberately want the run recorded in the table.

(3) Evaluate

python mmgr/embodied_navigation/vlm_based_evaluation/veo_evaluation.py \
    --input_dir       ./outputs/embodied/task01_veo3 \
    --source_data_dir ./data/image/embodied_navigation \
    --prompt_file     mmgr/embodied_navigation/panoramic_view_last-mile_navigation/vlm_based_evaluation/gemini2.5-pro_prompt.py \
    --output_dir      ./eval/embodied/task01_veo3

python mmgr/embodied_navigation/vlm_based_evaluation/nano_banana_evaluation.py \
    --input_dir       ./outputs/embodied/task01_nano_banana \
    --source_data_dir ./data/image/embodied_navigation \
    --prompt_file     mmgr/embodied_navigation/panoramic_view_last-mile_navigation/vlm_based_evaluation/image_gen_evaluation_prompt.py \
    --output_dir      ./eval/embodied/task01_nano_banana

Both evaluators can infer --source_data_dir, --prompt_file and --output_dir from the input directory layout; pass them explicitly when your layout differs from the released one. run_all_image_evaluations.sh batches all four tasks for the image models.

D3 · Physical Commonsense

50 instances: 25 VideoPhy examples spanning solid–solid, solid–fluid and fluid–fluid interactions (T9), and 25 compositional sports scenarios — ballet, skiing, diving, swimming — where motion depends on balance, momentum, projectile motion and fluid dynamics (T10). A successful generation must exhibit physics accuracy, motion quality, visual realism, and prompt adherence.

This domain is prompt-driven: there is no input image and no ground-truth solution image.

(1) Build the prompt sets (optional)

python mmgr/physical_commonsense/data_generation/generate_sports_prompts.py   # → sports_prompts.json
python mmgr/physical_commonsense/data_generation/sample_videophy_25.py        # → 25 VideoPhy samples

(2) Generate videos

python mmgr/physical_commonsense/generation/veo_generation.py \
    --data-dir   ./data/prompt/physical_commonsense \
    --output-dir ./outputs/physics/veo3 \
    --max-concurrent 3

(3) Evaluate

python mmgr/physical_commonsense/vlm_based_evaluation/veo_evaluation.py \
    --input_dir  ./outputs/physics/veo3 \
    --data_dir   ./data/prompt/physical_commonsense \
    --output_dir ./eval/physics/veo3

run_all_evaluation.sh runs every model in one pass.

Generating with Sora-2

Sora-2 is served through an Azure deployment rather than the Google stack, so it has its own client under mmgr/common/sora2/:

export AZURE_OPENAI_API_KEY=...

# The three entry points read three DIFFERENT variables, and two of them want a full
# API path rather than just the host. Export all three:
export AZURE_SORA_BASE_URL=https://YOUR-RESOURCE.openai.azure.com/openai/v1/video/generations/jobs
export AZURE_OPENAI_BASE_URL=https://YOUR-RESOURCE.openai.azure.com/openai/v1
export AZURE_OPENAI_ENDPOINT=https://YOUR-RESOURCE.openai.azure.com

# drive the job lifecycle explicitly
python mmgr/common/sora2/job_monitor.py create  --prompt "..." --width 1080 --height 1080 --seconds 5

JOB_ID="<paste the id printed above>"
python mmgr/common/sora2/job_monitor.py monitor --job-id "$JOB_ID" --timeout 300 --save
python mmgr/common/sora2/job_monitor.py list

# batch across a whole prompt set (checkpointing + retry) — the production path
python mmgr/common/sora2/batch_generate_parallel.py

# fetch every completed job's video
python mmgr/common/sora2/download_all_videos.py

sora_client.py is the reusable client class the other three build on. Four earlier one-off scripts that duplicated these paths were dropped during the open-source cleanup.

Evaluation protocol

The gated primary score

Each task's success criteria are decomposed into fine-grained metrics with three roles:

Role	Question it answers	Examples
Integrity / constraint	Did the model corrupt the problem?	maze unaltered, Sudoku clues unchanged, grid integrity, destination integrity, scene consistency
Outcome / correctness	Did it get the right answer?	target reached, grid completed accurately, correct final answer, success / oracle success
Reasoning process	Did it get there validly?	action reflection, no wall crossing, trajectory alignment, agent consistency

The primary score is gated: it equals 1 only when all required metrics hold, and 0 otherwise. An output that looks plausible but violates a single constraint still scores 0. This is deliberate — it is what stops "visually convincing" from being scored as "correct".

For video outputs, both the final outcome and, where relevant, the validity of intermediate frames are scored. For image outputs, the same task-level constraints are applied to the single generated output.

Full per-metric definitions are in docs/METRICS.md.

Deterministic evaluation

For tasks with exact ground truth, rule-based checks produce the headline numbers — no judge model is involved:

Maze — pixel-based path reconstruction. Frames are extracted, the green marker is tracked, all positions it visited are accumulated into the "generated path", and that path is compared against the reference: does it cross black walls, does the maze structure and start/goal placement still match, does the path match the reference under dilate + IoU, and is there backtracking behaviour.
Sudoku — OCR followed by exact-match and constraint checks: are the given clues preserved verbatim, and does the completed grid satisfy every row, column and box constraint.

For videos these checks are applied frame by frame; for images, to the single output.

A note on which evaluator produced the headline Maze/Sudoku numbers. The paper's main text and the Table 4 caption both state that Maze and Sudoku scores come from the deterministic evaluators, while two sentences in the appendix say the opposite (that the VLM overall is primary and the deterministic check is diagnostic). The reported numbers settle it: averaging the six pixel-based Overall values for Maze/Veo-3 gives exactly the 3.67 in Table 4, and the six OCR-based Overall values for Sudoku/Nano-banana give exactly 8.60 — neither is reproducible from the VLM tables. This repository therefore documents the deterministic evaluators as primary, and we flag the appendix wording as a known inconsistency in the paper rather than silently picking a side.

VLM-based evaluation

Gemini-2.5-Pro acts as the judge wherever there is no reliable pixel- or symbol-level verifier. It scores the intermediate and final criteria of a task-specific rubric, which then combine into the gated primary score.

Its coverage is wider than "the hard-to-verify domains" — be precise about which numbers depend on it:

Task	Primary score comes from	Gemini-2.5-Pro also used for
Maze	pixel evaluator (deterministic)	diagnostic fine-grained metrics
Sudoku	OCR evaluator (deterministic)	diagnostic fine-grained metrics
ARC	Gemini-2.5-Pro	—
Math	Gemini-2.5-Pro	—
All 4 navigation tasks	Gemini-2.5-Pro	—
Physical Concepts, Sports	Gemini-2.5-Pro	—

So 8 of the 10 tasks have a judge model in the primary path. Only Maze and Sudoku are judge-free.

The judge prompt, criteria file and output format for each task are checked into this repository alongside the runner, so the rubric is inspectable and modifiable rather than hidden inside a script.

Human evaluation

Three graduate-level students, paid $25/hour, independently scored balanced subsets — several dozen to several hundred outputs per task. Each output was judged by at least two annotators, with disagreements resolved by discussion. Annotators applied the same gated rubric as the automatic evaluators, saw the task input, the model output (video played frame by frame) and the ground-truth solution where one existed, and assigned a binary 0/1 label per criterion. Judgments follow literal task constraints, not overall visual quality.

How reliable is each number

This table bounds how much confidence each conclusion deserves. It is agreement between human annotations and the automatic evaluator.

Task	Evaluator	Model	Agreement acc. (%)
Maze	Pixel (deterministic)	Sora-2	100.00
Maze	Pixel (deterministic)	Veo-3	95.56
Maze	Pixel (deterministic)	Wan-2.2	100.00
Sudoku	OCR (deterministic)	Sora-2	100.00
Sudoku	OCR (deterministic)	Veo-3	100.00
Sudoku	OCR (deterministic)	Wan-2.2	100.00
ARC	VLM	Veo-3	94.90
3D R.-W. Nav.	VLM	Veo-3	75.33
3D R.-W. Nav.	VLM	Nano-banana Pro	21.67
Last-Mile Nav.	VLM	Veo-3	53.50
Last-Mile Nav.	VLM	Nano-banana Pro	30.00
Top-down Nav.	VLM	Veo-3	79.15
Top-down Nav.	VLM	Nano-banana Pro	51.69
SLAG	VLM	Veo-3	88.45
SLAG	VLM	Nano-banana Pro	62.39
Physics	VLM	Veo-3	65.31

Read this table before citing any single number.

Maze and Sudoku are the strongest evidence in the benchmark — deterministic checks, 95.56–100% agreement.
ARC's 94.90% is partly inflated by agreement on the many failed cases, not by fine-grained correctness.
Embodied navigation VLM agreement ranges from 21.67% to 88.45%. Exact model rankings in that domain should be read cautiously even where the aggregate trend is informative. The two Nano-banana Pro rows in particular are low enough that its navigation lead, while consistent across all four tasks, rests on a judge that agrees with humans less than half the time on two of them.
Physics reaches only 65.31%, so its scores are best read as judged physical plausibility rather than verified physical correctness.

Lower agreement bounds our confidence rather than rendering the results uninformative: absent a deterministic checker, VLM-based judgment remains the best available proxy. But the asymmetry is real, and MMGR reports it rather than averaging it away.

Evaluating your own model

The cleanest integration point is stage 2. Nothing in the evaluation stage is coupled to a particular generator — the evaluators consume a directory of media files and pair them against the benchmark inputs.

Read the benchmark inputs for your task (problem images, or the prompt JSON for physical commonsense; for navigation, the taskNN_test_table.json manifest).
Write your model's outputs into a directory using the same per-sample naming as the released generation outputs — *_solution.png for images, .mp4 for video, preserving the difficulty subdirectory structure.
Point the task's evaluator at that directory with --input_dir.

Draw five samples per prompt to match the protocol. For the deterministic Maze and Sudoku evaluators, --model_name is only a label used in the output filenames, so any string works.

If your model needs a different prompt phrasing, the prompt templates are plain files (generation_prompts/*.py, evaluation_prompt.txt) — copy and edit them rather than patching the runners. Note that changing the generation prompt makes your numbers non-comparable to the paper's, which used one standardized prompt per task across all models.

Cost and compute

Reproducing MMGR is not cheap. The table below is the paper's accounting for the entire reported experiment — all eight generative models, both Gemini baselines and the judge — not the cost of running one model through the benchmark:

Model	Backend	Est. cost (USD)
Veo-3	API	~30k
Sora-2	API	~9k
Four closed image models	API	~1.8k
Gemini baselines + Gemini-2.5-Pro judge	API	~3k
Wan-2.2	A100, ~1,340 h	~2.0k
Qwen-Image	A100, ~100 h	~0.15k
Total		~46k

Open-source throughput on a single A100 80GB: Qwen-Image renders one image in ~40 s; Wan-2.2 produces one clip in ~520 s. That is ~100 and ~1,340 A100 GPU-hours respectively, ~1,440 total.

Generation volume per model, at five samples per instance: a video model produces 9,265 generations; an image model 9,015 (the eight non-physics tasks); a Gemini text baseline 4,335 (Maze, Sudoku and Math only).

Assumed prices in that estimate: ~$1.5 per A100 GPU-hour; ~$0.05 per image; $0.40 and $0.10 per second of generated video for Veo-3 and Sora-2. The deterministic Maze and Sudoku checks run on CPU at negligible cost.

If you only want a signal, not the full benchmark, run Maze and Sudoku first: they are the cheapest to evaluate (CPU-only, no judge tokens) and carry the strongest evidence.

Known limitations

Stated plainly, because they bound what MMGR's numbers mean:

Zero-shot only. Fine-tuned or task-specific models are excluded by design. This isolates pre-trained reasoning ability but may underestimate what task adaptation could reach.
Text-to-video and text-to-image only. Video-to-video editing and other conditioning modalities are an orthogonal capability MMGR does not assess.
Deterministic verification covers only two tasks. Maze and Sudoku have exact checks. The primary scores for Physical Commonsense and for the embodied navigation tasks come from VLM judgment, inheriting VLMs' known weaknesses in spatial reasoning and physical understanding. (Some navigation subsets do have deterministic diagnostics; it is the gated primary score that is judge-based.)
The model pool misses recent systems. It covers leading video and image generators, but the pace of releases means some newer systems are absent.
Floor effects. Sudoku and ARC are near-floor for current video models, which limits discriminative power among weaker systems.
Human-designed difficulty tiers may not match model-specific difficulty gradients.
Prompt sensitivity. Standardized prompts prioritize cross-model consistency over per-model optimization; a differently-phrased prompt could move numbers.
Structured tasks, not open worlds. Finite mazes, fixed Sudoku grids and standardized math problems do not capture ambiguous objectives, partial observability, or open-ended creative generation. The benchmark also leaves out continuous control and long-horizon planning entirely.
Ecological validity is bounded. Because video models are trained on internet-scale data, they may perform well on distribution-matched scenarios that these controlled tasks omit — so MMGR gives an incomplete picture of their capabilities, not an upper bound on them.
Sampled reliability analysis. The agreement study rests on sampled manual annotation, not exhaustive labeling. Raw accuracy can be inflated by class imbalance and does not substitute for fine-grained agreement metrics. Because annotating every generated output is infeasible, comprehensive assessment still depends on the automatic metrics — especially for the VLM-only tasks.

FAQ

Do I need to run data generation? No. Download the released data from HuggingFace and start at stage 2. Data generation is provided for transparency and for anyone who wants to change the difficulty distribution.

Why do maze and sudoku have two evaluators each? The deterministic one (pixel / OCR) produces the headline numbers because it agrees with humans 95.56–100% of the time. The VLM evaluator is kept for diagnostic comparison and for fine-grained metrics the deterministic checker does not produce.

Why do embodied navigation solutions ship as 0 files? The source ground-truth mapping still needs approval before publication. Generation and VLM-based evaluation both work without them.

Why are task01–task04 still in the code when the paper says T5–T8? They are the manifest identifiers, and renaming them would break the released test tables. Note the two spellings are not interchangeable: --task takes task_01–task_04 (underscore), while the manifest files are named task01_test_table.json (no underscore). The mapping to T5–T8 is in Repository structure.

Can I evaluate an image model on Physical Commonsense? The paper does not — that domain was run on video models only, which is why those columns are "–" in the results table. Nothing in the code prevents it, but you would be reporting a new setting, not reproducing a paper number.

The math evaluator wants both an image dir and a video dir. Can I pass only one? Both --image_results_dir and --video_results_dir are required. Point the unused one at an empty directory.

Citation

@inproceedings{cai2026mmgr,
  title     = {{MMGR}: Multi-Modal Generative Reasoning Benchmark and Evaluation},
  author    = {Cai, Zefan and Qiu, Haoyi and Ma, Tianyi and Zhao, Haozhe and
               Zhou, Gengze and Liao, Tingting and Yu, Xinyan Velocity and
               Huang, Kung-Hsiang and Wan, Ke and Lin, Shawn and
               Kordjamshidi, Parisa and Zhang, Minjia and Xiao, Wen and
               Gu, Jiuxiang and Peng, Nanyun and Hu, Junjie},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year      = {2026}
}

Authors

Zefan Cai^1,*, Haoyi Qiu^2,*, Tianyi Ma^3,*, Haozhe Zhao^4,*, Gengze Zhou⁵, Tingting Liao⁸, Xinyan Velocity Yu⁹, Kung-Hsiang Huang⁶, Ke Wan, Shawn Lin, Parisa Kordjamshidi³, Minjia Zhang⁴, Wen Xiao⁷, Jiuxiang Gu¹⁰, Nanyun Peng², Junjie Hu¹

¹ University of Wisconsin–Madison · ² University of California, Los Angeles · ³ Michigan State University · ⁴ University of Illinois Urbana–Champaign · ⁵ University of Adelaide · ⁶ Salesforce AI Research · ⁷ Microsoft · ⁸ MBZUAI · ⁹ University of Southern California · ¹⁰ Adobe Research

^* Equal contribution.

Acknowledgements

MMGR builds on data and tooling from ARC-AGI (fchollet/ARC, michaelhodel/re-arc, google/ARC-GEN), GSM8K, MATH-500, AIME 2024/2025, Omni-MATH, Matterport3D, HM3D, and VideoPhy. We thank their authors.

The physical-commonsense task samples from VideoPhy; the VideoPhy scoring model itself is not vendored here — see the upstream repository if you want to run it.

Contact

Zefan Cai (zefncai@gmail.com) · Haoyi Qiu (haoyiqiu@g.ucla.edu) · Tianyi Ma (matiany3@msu.edu) · Haozhe Zhao (haozhez6@illinois.edu)

Issues and pull requests are welcome.

MMGR reframes evaluation of multimodal generation from whether outputs look realistic to whether they solve the underlying reasoning problem.

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation

Table of Contents

What is MMGR

Why another benchmark

Benchmark at a glance

Headline results

Repository structure

Installation

Credentials

Getting the data

The three-stage pipeline

Reproducing each task

D1 · Abstract Reasoning

T1 · Maze

T2 · Sudoku

T3 · ARC

T4 · Math

D2 · Embodied Navigation

D3 · Physical Commonsense

Generating with Sora-2

Evaluation protocol

The gated primary score

Deterministic evaluation

VLM-based evaluation

Human evaluation

How reliable is each number

Evaluating your own model

Cost and compute

Known limitations

FAQ

Citation

Authors

Acknowledgements

Contact