MMGR Benchmark

August 25, 2026 · View on GitHub

All numbers in this file are taken verbatim from the MMGR paper appendix.

MMGR (Multimodal Generative Reasoning) comprises 10 tasks across 3 domains, totalling 1,853 evaluation task instances (Table 1 / Section A.1). Each generative model draws five outputs per prompt.

Overview table (task × reasoning ability × sample count)

The paper's Section 4 lists five core reasoning abilities: Physical, Logical, 2D Spatial, 3D Spatial, Temporal.

Fidelity note. The appendix digest used as the source for this document does not reproduce the exact checkmark grid of the paper's Table 1. The mapping below is compiled from the per-task descriptions in the appendix. Legend: ✓ = the ability is explicitly named for that task in the appendix text; (✓) = the ability is directly entailed by the task's own definition/metrics (e.g. 3D geometric success ⇒ 3D Spatial) but is not printed as a named "reasoning ability" in the digest; — = not indicated.

Task	Name	Domain	Physical	Logical	2D Spatial	3D Spatial	Temporal	Samples
T1	Maze	D1 Abstract Reasoning	—	(✓)	(✓)	—	—	240
T2	Sudoku	D1 Abstract Reasoning	—	✓	—	—	—	300
T3	ARC	D1 Abstract Reasoning	—	✓	✓	—	—	456
T4	Math	D1 Abstract Reasoning	—	✓	✓	—	✓	327
T5	Panoramic View Last-Mile Navigation	D2 Embodied Navigation	(✓)	—	—	(✓)	—	120
T6	Top-down View Navigation	D2 Embodied Navigation	(✓)	—	(✓)	—	—	120
T7	3D Real-World Navigation	D2 Embodied Navigation	(✓)	—	—	(✓)	—	120
T8	SLAG (Simultaneous Localization and Generation)	D2 Embodied Navigation	(✓)	—	(✓)	(✓)	—	120
T9	Physical Concepts	D3 Physical Commonsense	✓	—	—	—	(✓)	25
T10	Sports	D3 Physical Commonsense	✓	—	—	—	(✓)	25
		Total						1,853

Per-domain totals (Section A.1):

Domain	Tasks	Instances
D1 Abstract Reasoning	Maze (240), Sudoku (300), ARC (456), Math (327)	1,323
D2 Embodied Navigation	Last-Mile (120), Top-down (120), 3D R.-W. (120), SLAG (120)	480
D3 Physical Commonsense	Physical Concepts (25), Sports (25)	50
Total		1,853

T1 — Maze

Task ID: T1
Paper name: Maze (2D Maze task) — Appendix C
Domain: D1 Abstract Reasoning
Reasoning ability probed: The appendix does not print a named reasoning-ability list for Maze; it is a single-solution pathfinding task on a 2D grid (Logical / 2D Spatial by construction).
Sample count: 240
Data source (dataset + version + generation tool):
- Generated with the open-source Python library maze-dataset (Ivanitskiy et al., 2023). Library version: not specified in paper.
- Two maze-generation algorithms: Depth-First Search (DFS) and Wilson's Algorithm.
- Each maze has exactly one solution path (unique solution guaranteed).
Difficulty grading dimensions and per-bucket counts: Complexity controlled along three axes.
- Generators (2): DFS, Wilson's Algorithm.
- Grid Sizes (10 levels): from 3×3 to 13×13.
- Start–Goal Placement (4 schemes): corner→corner (c→c), corner→random (c→r), random→corner (r→c), random→random (r→r). A minimum start–goal distance is enforced.
- Difficulty buckets (grid-size): Easy 3×3–5×5; Medium 6×6–9×9; Hard 10×10–13×13.
- Per bucket: each generator produces 120 mazes = 40 Easy + 40 Medium + 40 Hard. Two generators × 120 = 240 total.
Input format: A maze rendering with a green square (start), a red square (end/target), and black walls (white corridors). The generation prompt / template used to elicit a solution is not specified in paper for Maze.
Expected output format: A video (video models) or a single image (image models) in which the green agent traverses the maze from start to the red target along a valid path. For image outputs the solution is a blue path connecting start and end. No train/val/test split described.

T2 — Sudoku

Task ID: T2
Paper name: Sudoku — Appendix D
Domain: D1 Abstract Reasoning
Reasoning ability probed: Constraint Satisfaction and Logical Reasoning (logical inference / deductive reasoning); the model must ensure each symbol appears exactly once in every row, column, and subgrid (Seely et al., 2025).
Sample count: 300
Data source (dataset + version + generation tool):
- Puzzles are programmatically generated by the authors. Generator library/package name and version: not specified in paper.
- Sudoku-bench (Seely et al., 2025) is cited as motivation, not as a generation tool.
- OCR/evaluation tooling: PaddleOCR (adaptive thresholding, denoising, cell segmentation); version: not specified in paper.
Difficulty grading dimensions and per-bucket counts: Two primary axes.
- Grid Size (2): 4×4 (digits 1–4); 9×9 (digits 1–9).
- Puzzle Difficulty (3): Easy (many clues), Medium, Hard (few clues while keeping a unique solution) — controlled by the number of initial clues.
- Per bucket: for every difficulty level (Easy, Medium, Hard) 100 puzzles are generated: 50 for 4×4 and 50 for 9×9. So each (grid size × difficulty) bucket = 50 puzzles; 6 buckets × 50 = 300.
Input format: A rendered Sudoku puzzle image (4×4 or 9×9) with pre-filled clue digits. Generation prompt template: not specified in paper.
Expected output format: A video (video models) or image (image models) showing the fully completed grid; clues must remain intact, all row/column/subgrid constraints satisfied, and all empty cells filled correctly. No train/val/test split described.

T3 — ARC

Task ID: T3
Paper name: ARC (Abstraction and Reasoning Corpus) — Appendix E
Domain: D1 Abstract Reasoning
Reasoning ability probed: Abstract Reasoning, Pattern Recognition, Rule Induction; jointly tests 2D spatial reasoning (grid-based patterns) and logical reasoning (deducing/executing abstract transformations) (Chollet, 2019).
Sample count: 456
Data source (dataset + version + generation tool):
- Built on the open-source ARC benchmark (Chollet, 2019). 456 tasks from two benchmark versions:
  - ARC v1: 381 tasks (the publicly released ARC training set).
  - ARC v2: 75 tasks (newly added, novel pattern families, higher complexity).
- Cited repos / toolkit (footnote 3): https://github.com/fchollet/ARC, https://github.com/michaelhodel/re-arc, and generation toolkit https://github.com/google/ARC-GEN. Specific tool parameters: not specified in paper.
- All tasks manually curated. Transformation categories: symmetry, rotation, scaling, color manipulation, object-level reasoning.
Difficulty grading dimensions and per-bucket counts: Two-Level Classification System.
- Level 1 — Shape Consistency: Match (316 cases) = output grid same dimensions as input (in-place transforms); Mismatch (140 cases) = output dimensions differ (cropping, extraction, concatenation, reconstruction; generally harder).
- Level 2 — Quantitative Difficulty (Easy/Medium/Hard): sum of five grid-level feature scores — Grid Size (min side length), Color Count, Object Count, Occupancy Ratio, ∆IO (grid change ratio; Match cases only). Thresholds: Match cases (5 features) Easy ≤3, Medium 4–6, Hard ≥7; Mismatch cases (4 features) Easy ≤2, Medium 3–4, Hard ≥5.
- Per-bucket counts (overall): Easy 137, Medium 215, Hard 104 (total 456). By shape consistency: Match — Easy 103, Medium 151, Hard 62 (316); Mismatch — Easy 34, Medium 64, Hard 42 (140). (Full v1/v2 split in Table 11, see RESULTS.md.)
Input format: ARC-style demonstration examples (input→output grid pairs) plus a test input grid. Generation prompt template: not specified in paper.
Expected output format: A generated video or image whose predicted output grid should exactly match the ground-truth output grid. No train/val/test split described (the 456 cases are the evaluation suite).

T4 — Math

Task ID: T4
Paper name: Math (Visual Math task) — Appendix F
Domain: D1 Abstract Reasoning
Reasoning ability probed: Logical Reasoning, Logical Deduction, and 2D Spatial Reasoning; video generation additionally challenges Temporal Reasoning (animating step-by-step deduction from premises to solution) (Huang et al., 2025a).
Sample count: 327
Data source (dataset + version + generation tool): Five benchmarks.
- GSM8K (Cobbe et al., 2021) — grade school word problems — 50.
- MATH / MATH500 (Hendrycks et al., 2021) — high school competition math — 50 (Table 19 labels the split "MATH500").
- AIME 2024 (Mathematical Association of America, 2024) — 30.
- AIME 2025 (Mathematical Association of America, 2025) — 30.
- Omni-MATH (Gao et al., 2025) — Olympiad-level — 167.
- Additional construction tooling: not specified in paper.
Difficulty grading dimensions and per-bucket counts:
- Per-benchmark counts (Table 19): GSM8K 50, MATH500 50, AIME 2024 30, AIME 2025 30, Omni-MATH 167 → Total 327.
- Omni-MATH difficulty (5 levels): T0 (easiest, middle school) → T4 (hardest, Olympiad).
- Omni-MATH category (8 types): Algebra, Applied Math, Calculus, Discrete Math, Geometry, Precalculus, Number (Number Theory), Other.
- Omni-MATH distribution (Table 20 category totals): Algebra 24, Applied Math 25, Calculus 20, Discrete Math 25, Geometry 24, Precalculus 23, Number 24, Other 2 (sum = 167). Per difficulty×category cells given in Table 20 (see RESULTS.md).
Input format: A mathematical problem presented visually (e.g. geometry, diagrams). Generation prompt template: not specified in paper.
Expected output format: A generated video (video models) or image (image models) animating / showing the step-by-step deduction and final answer. No train/val/test split described.

Task ID: T5
Paper name: Panoramic View Last-Mile Navigation (Last-Mile Nav.) — Appendix G (shared) + H
Domain: D2 Embodied Navigation
Reasoning ability probed: Not printed as a named list in the digest; the task presents a 360° panoramic third-person ("over-the-shoulder") environment and isolates precise localization of a visible goal and generation of an optimal short-horizon trajectory (3D spatial + physical validity by construction).
Sample count: 120
Data source (dataset + version + generation tool):
- Scenes from Matterport3D (Chang et al., 2017) and HM3D (Ramakrishnan et al., 2021), plus rendered environments in Habitat (Savva et al., 2019). Versions / render parameters: not specified in paper.
- Ground-truth traces for the Geo-Align pipeline: shortest-path coordinates recorded via manual control in Habitat; generated sequences reconstructed with VGGT (Wang et al., 2025a).
Difficulty grading dimensions and per-bucket counts: Four shared hard-level axes → 24 configurations, 5 samples each = 120 (Table 25, uniform).
- Environmental Complexity (2): floor01 (single floor, no vertical transitions); floor02plus (multi-level, implicit/explicit stairs).
- View Fidelity (3): scored 3–5 → quality03, quality04, quality05 (here: extent/spatial distribution of occlusions).
- Trajectory Distance (2): short (direct motion, no major turns, may include vertical movement); long (≥1 significant turn; shares partial structure with matching short case).
- Destination Specification (2): color mark (pure red #ff0000 overlay); location description (natural-language description with disambiguating details).
- (Table 28 Geo-Align uses diagnostic groupings noturn/oneturn and color/object; per-bucket counts for that table are not specified in paper.)

Input format: A single panoramic RGB observation; target given either by a red marker or an object-class description. Generation prompt template (Figure 21, verbatim, incl. original typo "to navigates"):

Prompt Text:
    Create a photorealistic video depicting a humanoid robot performing "last mile" navigation. The robot
must traverse a static, 3D panoramic environment populated with obstacles (furniture/walls) to navigates
directly toward a bright red (#ff0000) target area marked on the floor. The camera must move smoothly to
follow the robot, in a Third-person, "over-the-shoulder" view, adjusting pan and tilt to keep both the agent
(foreground) and the red target (background) continuously in frame.

Expected output format: A video (or image) of the humanoid robot navigating to the goal. No train/val/test split described.

Task ID: T6
Paper name: Top-down View Real-World Navigation (Top-down View Nav.) — Appendix G (shared) + I
Domain: D2 Embodied Navigation
Reasoning ability probed: Advanced spatial reasoning with semantic understanding; interpret partially occluded top-down layouts (floor plans) from a fixed bird's-eye camera and ground goals from textual descriptions of varying abstraction (2D spatial planning + physical validity by construction).
Sample count: 120
Data source (dataset + version + generation tool):
- Scenes from Matterport3D, HM3D, and Habitat (same shared sources; versions/params not specified in paper).
- Multi-floor layouts may include basements, attics, outdoor pools, gardens. GT sequences recorded from synchronized fixed-camera viewpoints in the simulator. Start = blue triangle, goal = red area.
Difficulty grading dimensions and per-bucket counts: Same four shared axes → 24 configurations × 5 = 120 (Table 25).
- Environmental Complexity (2): 1 Floor / 2 Plus Floors.
- View Fidelity (3): quality03/quality04/quality05 (holistic scene realism/navigability: holes/cracks, furnishing, door openness, interaction plausibility; 3–5 scale).
- Trajectory Distance (2): short/long (Table 29 reports these as noturn/oneturn).
- Destination Specification (2): color mark (#ff0000) / location description.

Input format: A fixed bird's-eye top-down layout image; the robot starts on a bright blue (#0000ff) triangular marker; target is a red (#ff0000) area. Generation prompt template (Figure 23, verbatim):

Prompt Text:
    Create a video showing an agent navigating a 2D top-down environment. The robot begins positioned on
a bright blue (#0000ff) triangular marker, with its front aligned to the triangle's pointing vertex. It travels
smoothly and efficiently toward a static red (#ff0000) target area. As the robot agent moves, it leaves a
permanent, bright green (#00ff00) trajectory line tracing its exact path from the start point. The camera is
single, strictly static and fixed top-down view that captures the full environment.

Expected output format: A video (or image) with a green (#00ff00) trajectory line tracing the agent's path from start to the red goal under a static top-down camera. No train/val/test split described.

Task ID: T7
Paper name: 3D Real-World Navigation (3D R.-W. Nav.) — Appendix G (shared) + J
Domain: D2 Embodied Navigation
Reasoning ability probed: Interpret egocentric visual streams and parse complex 3D environmental geometry; ground navigation decisions in multi-room, multi-level layouts from a fixed third-person cutaway/"dollhouse" view (3D spatial + physical validity by construction).
Sample count: 120
Data source (dataset + version + generation tool):
- Matterport3D, HM3D, Habitat (Chang et al., 2017; Ramakrishnan et al., 2021; Chaplot et al., 2020b / Savva et al., 2019). Renderer versions/params: not specified in paper.
- Environment View: 3D (cutaway/dollhouse); Camera View: fixed third-person. Multi-floor layouts include basements, attics, pools, gardens. GT = shortest-path from synchronized fixed-camera viewpoints.
Difficulty grading dimensions and per-bucket counts: Same four shared axes → 24 configurations × 5 = 120 (Table 25).
- Environmental Complexity (2): 1 Floor (floor01) / 2 Plus Floors (floor02plus) — 60 / 60.
- View Fidelity (3): quality03/quality04/quality05 — 40 / 40 / 40.
- Trajectory Distance (2): short/long — 60 / 60.
- Destination Specification (2): color mark/location description — 60 / 60.
- Diagnostic-subset size for Table 31: not specified in paper.

Input format: A cutaway/dollhouse 3D rendering with a red (#ff0000) goal region on the floor; requires valid action trajectories (e.g. move forward, turn left). Generation prompt template (Figure 25, verbatim):

Prompt Text:
    Create a video showing a humanoid robot successfully navigating from a starting point through a multi-
story indoor space. The robot obeys physical constraints and is the only object that moves in the entire
video, beginning standing at the designated starting location on the floor toward a fixed bright red
(#ff0000) region marked on the floor. When moving between floors, the robot climbs or descends the
staircase step-by-step, keeping its feet planted on individual treads. The camera is third-person isometric
view (a "dollhouse" view).

Expected output format: A video (or image) of the robot navigating (including stair climbing/descending) to the red goal under a fixed third-person isometric/dollhouse camera. No train/val/test split described.

T8 — SLAG (Simultaneous Localization and Generation)

Task ID: T8
Paper name: SLAG (Simultaneous Localization and Generation) — Appendix G (shared) + K
Domain: D2 Embodied Navigation
Reasoning ability probed: Jointly localize the agent and generate the surrounding scene layout; achieve precise spatiotemporal alignment between physical 3D navigation and generative 2D plotting (extends SLAM; 2D + 3D spatial + physical validity by construction).
Sample count: 120
Data source (dataset + version + generation tool):
- Matterport3D, HM3D, Habitat (versions/params not specified in paper).
- Environment View: 3D + Top-down; Camera View: fixed third-person. The 2D map is provided but the destination is not; 2D success is conditioned on the same 3D destination. Start = robot position / blue triangle; goal = red area. GT = shortest-path from synchronized fixed-camera viewpoints.
Difficulty grading dimensions and per-bucket counts: Same four shared axes → 24 configurations × 5 = 120 (Table 25).
- Environmental Complexity (2): 1 Floor/2 Plus Floors — 60 / 60.
- View Fidelity (3): quality03/quality04/quality05.
- Trajectory Distance (2): short/long.
- Destination Specification (2): color mark (#ff0000)/location description.

Input format: A 3D indoor "dollhouse" scene plus a fixed 2D top-down map; a bright red (#ff0000) target painted on the floor in 3D. Generation prompt template (Figure 27, verbatim):

Prompt Text:
    Create a split-screen video for the task "Simultaneously Localization and Generation." The left panel
must show a humanoid robot navigating a photorealistic 3D indoor scene to a bright red (#ff0000) target
area painted on the floor, while the right panel simultaneously shows a generated bright green (#00ff00)
trajectory line that grows as the robot advances in a fixed 2D top-down map in real time. Keep the
camera static, isometric, and wide enough to include the entire navigable layout.

Expected output format: A split-screen video (3D navigation on the left, growing green 2D top-down trajectory on the right). No train/val/test split described.

T9 — Physical Concepts

Task ID: T9
Paper name: Physical Concepts (Atomic / Fundamental Interactions) — Appendix L
Domain: D3 Physical Commonsense
Reasoning ability probed: Fundamental physical principles (statics, dynamics, kinematics) in controlled environments; Physical understanding with temporal/causal interactions over time.
Sample count: 25
Data source (dataset + version + generation tool):
- Structured ontology adopted from VideoPhy (Bansal et al., 2025) and VideoPhy-2 (Bansal et al., 2026).
- Captioned interaction pool: Solid–Solid (rigid collisions, stacking), Solid–Fluid (splashing, buoyancy), Fluid–Fluid (diffusion, mixing).
Difficulty grading dimensions and per-bucket counts: Shared D3 stratification along three dimensions.
- Interaction Type ("What"): Solid-Solid, Solid-Fluid, Fluid-Fluid (Table 38 additionally reports an "Action/Other" category).
- Scenario Context ("Where"): controlled physics experiments (Physical Concepts) vs unconstrained dynamic environments (Sports).
- Interaction Complexity ("How"): Simple, Complex, Chain-Reaction.
- Difficulty levels reported for T9 (Table 38): Easy and Hard (Medium not broken out).
- Per-bucket sample counts: not specified in paper (only the total of 25 and per-category success rates are given).
Input format: A three-field per-scenario schema (Prompt Text / Physics Focus / Expected Motion), e.g. Prompt Text: Metal grinder crushing coffee beans. / Physics Focus: States of matter interaction: solid_solid / Expected Motion: Natural interaction between materials. Full generation prompt template: not specified in paper.
Expected output format: A generated video (evaluation is exclusive to video generative models — Veo-3, Sora-2, Wan-2.2 — since static image generators cannot represent causal interactions over time). No further train/test split.

T10 — Sports

Task ID: T10
Paper name: Sports (Sports Scenarios / Compositional Contexts) — Appendix L
Domain: D3 Physical Commonsense
Reasoning ability probed: Compositional reasoning through dynamic, high-velocity human movements requiring simultaneous integration of multiple physical laws; Physical understanding with temporal/causal dynamics.
Sample count: 25
Data source (dataset + version + generation tool):
- A self-synthesized complementary dataset of sports-oriented prompts (no external dataset citation).
- Categories: Precision & Arts (e.g. ballet pirouettes / angular momentum), Winter Sports (e.g. skiing moguls / friction + gravity), Aquatics (e.g. diving, swimming / fluid resistance), Athletics. Table 37 enumerates sport types as Ballet, Diving, Skiing, Swimming.
Difficulty grading dimensions and per-bucket counts: Shared D3 stratification (see T9 / METRICS.md).
- Sport-type buckets (Table 37): Ballet, Diving, Skiing, Swimming.
- Difficulty levels reported for T10 (Table 37): Easy, Medium, Hard.
- Per-category / per-bucket sample counts: not specified in paper (only the total of 25 and per-category success rates are given).
Input format: Three-field per-scenario schema (Prompt Text / Physics Focus / Expected Motion), e.g. Prompt Text: A ballet dancer performs a fouetté turn... / Physics Focus: Conservation of angular momentum, centripetal force / Expected Motion: Rapid leg whip, continuous rotation, stable center. Full generation prompt template: not specified in paper.
Expected output format: A generated video (video generative models only). No further train/test split.