MMGR Benchmark
August 25, 2026 · View on GitHub
All numbers in this file are taken verbatim from the MMGR paper appendix.
Every fine-grained metric in MMGR is binary (0/1) unless otherwise noted (Constraint Violation and Completion Accuracy are fractions in [0,1]). Each task derives one gated primary ("Overall") score from its fine-grained metrics. The tables in the paper's Section 4 report the primary score per task; the fine-grained metrics are diagnostic.
Metric roles (Table 3 classification)
The paper's Table 3 color-codes metrics into three roles. Fidelity note: the appendix digest does not reproduce Table 3's exact color grid; the role assignment below is derived from each metric's described function and is labelled as such.
- integrity-constraint — a failure mode / preservation check: the model must not alter the given input or violate a hard constraint (e.g. Maze Changed, Cross Wall, Clues Changed, Constraint Violation, Grid Integrity, Destination Integrity, Scene Consistency, Object Semantic, Agent Consistency, Spatial Alignment, Physics Accuracy, Visual Realism).
- outcome-correctness — did the output reach the correct answer / goal (e.g. Target Achievement, Completion Accuracy, Valid Solution, Final Correctness, Pattern Recognition, Color Accuracy, Success Score 2D/3D, Oracle Success Score 2D/3D, Prompt Adherence).
- reasoning-process — evidence of the step-by-step / exploratory reasoning process (e.g. Action Reflection, Intermediate Correctness, Trajectory Alignment, Motion Quality).
Direction: ↓ lower is better (failure modes), ↑ higher is better.
T1 — Maze
Two evaluation pipelines: (A) VLM-based (C.3) — the source of the Table 4 Maze row — and (B) pixel-based deterministic (C.5.3) — a stricter diagnostic. The VLM evaluator is asked: (i) does the green square reach and stop on the red square? (ii) does the green square touch/cross a black wall? (iii) does the wall layout or red-square position change?
(A) VLM-based fine-grained metrics (rule-computed from VLM-judge yes/no answers)
| Metric | Meaning | Method | Values | Role | Gating |
|---|---|---|---|---|---|
| Maze Changed (MC) ↓ | Whether the maze layout is altered. Video: 1 if structure changes in any frame. Image: 1 if structure differs from reference solution. | VLM judge | 0/1 | integrity-constraint | Required = 0 |
| Cross Wall (CW) ↓ | Constraint violation. Video: 1 if green square touches/crosses a black wall in any frame. Image: 1 if the blue path touches/crosses walls. | VLM judge | 0/1 | integrity-constraint | Required = 0 |
| Action Reflection (AR) ↑ | Exploratory behaviour. Video: 1 if trajectory shows exploration (backtracking / multiple paths); 0 if single direct route. Image: N/A. | VLM judge | 0/1 (video only) | reasoning-process | Not gating |
| Target Achievement (TA) ↑ | Task completion. Video: 1 if green square reaches and stops on the red target. Image: 1 if a continuous valid blue path connects start and end. | VLM judge | 0/1 | outcome-correctness | Required = 1 |
| Overall ↑ | Binary success. | rule | 0/1 | primary | See below |
Gated primary (VLM, verbatim): "Set to 1 if MC = 0, CW = 0, and TA = 1; 0 otherwise." Action Reflection does not gate. The Table 4 Maze row is aggregated from this VLM-based Overall metric.
(B) Pixel-based deterministic metrics (classical CV; no VLM/LLM) — see "Deterministic evaluation" section
Four metric classes: Wall Crossing Detection, Layout Matching, Path Matching, Reflection/Exploration Detection.
- Image gating (verbatim): "maze match IoU > 0.8, no wall crossing detected, target achievement > 0.95, and reflection detected if the generated path contains loops or its length exceeds 1.3× the solution path length." Image gated overall = 1 iff layout correct AND no wall crossings AND sufficient target achievement (with no excessive reflection); else 0.
- Video gating (verbatim): same as images except layout matching is the minimum IoU across all frames and wall crossing is flagged only if the crossing ratio exceeds 5%. Video gated overall = 1 only when all conditions hold simultaneously (correct layout across all frames, no wall crossings, near-complete target coverage, no excessive reflection).
T2 — Sudoku
Two pipelines: (A) VLM-based (D.2 / D.4.1) — stated in D.4.1 as the source of the Table 4 Sudoku row — and (B) OCR-based deterministic (D.4.2, PaddleOCR). (Note: the paper's front-matter also describes the Sudoku score as OCR-based; the digest records both statements and flags the tension.) The VLM evaluator receives the generation, the ground-truth solved grid, and a structured prompt.
Fine-grained metrics (same set for both pipelines)
| Metric | Meaning | Method | Values | Role | Gating |
|---|---|---|---|---|---|
| Clues Changed (CC) ↓ | Whether any pre-filled clue is modified/removed/displaced. Video: 1 if any clue changes in any frame. Image: 1 if any given clue differs from the original. | VLM judge (A) / OCR rule (B) | 0/1 | integrity-constraint | Required = 0 |
| Constraint Violation (CV) ↓ | Fraction of row/column/subgrid constraints violated in the final output (0 = full compliance). | VLM judge (A) / OCR rule (B) | fraction [0,1] | integrity-constraint | Required = 0 |
| Completion Accuracy (CA) ↑ | Fraction of originally empty cells filled correctly vs ground-truth solution. | VLM judge (A) / OCR rule (B) | fraction [0,1] | outcome-correctness | Required = 1 (perfect) |
| Action Reflection (AR) ↑ | Video: 1 if the video shows explicit reflection (correcting earlier entries); 0 if updates appear simultaneously / non-reflective. Image: N/A. | VLM judge (A) / rule (B) | 0/1 (video only) | reasoning-process | Not gating |
| Overall ↑ | Perfect-solution success. | rule | 0/1 | primary | See below |
Gated primary (verbatim): "1 only if Clues Changed=0 AND Constraints Violation=0 AND Completion Accuracy=1; 0 otherwise." Identical for both pipelines; Action Reflection is not required for correctness.
OCR video Action Reflection (deterministic, D.4.2): verifies gradual filling — average ≤3 cell updates per frame for 4×4 (≤5 for 9×9), no more than 25% of cells changing in any single frame, and at least 20% of frames exhibiting non-zero updates.
T3 — ARC
Evaluated by a VLM taking four inputs: demonstration examples, test input, ground-truth output, and the generated video/image. All four metrics are VLM-judge based.
| Metric | Meaning | Method | Values | Role | Gating |
|---|---|---|---|---|---|
| Pattern Recognition ↑ | 1 if the model successfully identifies the transformation pattern from the demonstrations. | VLM judge | 0/1 | outcome-correctness (derived) | See note |
| Grid Integrity ↑ | 1 if the output preserves correct grid dimensions and structural layout; 0 if distorted/misaligned. | VLM judge | 0/1 | integrity-constraint | See note |
| Color Accuracy ↑ | 1 if all colors are applied correctly per the transformation rule. | VLM judge | 0/1 | outcome-correctness (derived) | See note |
| Valid Solution (Overall) ↑ | Primary metric: 1 only if the generated output exactly matches the ground-truth solution; 0 otherwise. | VLM judge | 0/1 | primary | Defined independently |
Gating: The paper does not state a multiplicative/AND formula gating Pattern Recognition × Grid Integrity × Color Accuracy into the primary score. The primary "Valid Solution / Overall" is defined independently as exact match. The empirical relationship is described as a "Metric Cascade" (verbatim): "Grid Integrity (55.79%) → Pattern Recognition (28.42%) → Color Accuracy (12.63%) → Overall (9.21%)" for Nano-banana on v1; "Color Accuracy emerges as the critical bottleneck, representing only 20–25% of Grid Integrity performance across all models." The aggregation formula from v1/v2 Overall into the single Table 12 Final score is not specified in paper (v1/v2 tables are diagnostic splits).
T4 — Math
Evaluated by Gemini-2.5-Pro (analyzes intermediate reasoning steps, final answers, reflective behaviour). All four metrics are VLM/LLM-judge based.
| Metric | Meaning | Method | Values | Role | Gating |
|---|---|---|---|---|---|
| Final Correctness (FC) ↑ / "Outcome Success Rate" | 1 if the final solution matches the ground-truth answer. | VLM judge | 0/1 | outcome-correctness | Required = 1 |
| Intermediate Correctness (IC) ↑ / "Process Success Rate" | 1 if all reasoning steps are logically valid and mathematically sound. | VLM judge | 0/1 | reasoning-process | Required = 1 |
| Action Reflection (AR) ↑ | Videos only: 1 if the video exhibits self-correction (revising steps / reconsidering approaches); 0 otherwise. N/A for images. | VLM judge | 0/1 (video only) | reasoning-process | Not gating |
| Overall (Overall Success Rate) ↑ | Primary metric: complete solution correctness. | VLM judge | 0/1 | primary | See below |
Gated primary (verbatim): "Overall Score: 1 if and only if both Final Correctness=1 AND Intermediate Correctness=1, 0 otherwise." I.e. Overall = FC ∧ IC. Action Reflection is diagnostic (video models) and does not gate. Result tables name these Process Success Rate (= IC), Outcome Success Rate (= FC), Action Reflection, and Overall Success Rate (= Overall).
T5–T8 — Embodied Navigation (shared metric suite, G.3)
All navigation metrics are binary (pass/fail = 1/0), computed directly from the agent's execution trace and generated frames. The Task Completeness (geometry) metrics are rule-based/geometric; Physical Understanding and Instruction Following metrics are VLM-judge (Gemini-2.5-Pro). Geo-Align (T5 only) and View-Syn (T6/T7/T8) are deterministic/rule-based distributional pipelines.
Task Completeness (geometry only)
| Metric | Meaning | Method | Role |
|---|---|---|---|
| Success Score 2D (S.S. 2D) ↑ | 1 if final coordinates fall entirely inside the goal footprint in the 2D overhead map; else 0. | rule (geometric) | outcome-correctness |
| Oracle Success Score 2D (O.S. 2D) ↑ | 1 if the path ever intersects/touches the 2D goal region (even without stopping); else 0. | rule (geometric) | outcome-correctness |
| Success Score 3D (S.S. 3D) ↑ | 1 if the final 3D position is within the target volume; else 0 (independent of visual discrepancies). | rule (geometric) | outcome-correctness |
| Oracle Success Score 3D (O.S. 3D) ↑ | 1 if the trajectory ever crosses the predefined proximity threshold around the target; else 0. | rule (geometric) | outcome-correctness |
| Trajectory Alignment Score (Traj. Ali.) ↑ | 1 if the 2D projected route strongly corresponds to the 3D motion path (major turns, spatial transitions); else 0. (Used by SLAG.) | rule (geometric) | reasoning-process |
Physical Understanding (VLM-judge, binary)
| Metric | Meaning | Role |
|---|---|---|
| Object Semantic Score (Obj. Sem.) ↑ | 1 if no collision/penetration with solid structures (walls, furniture, appliances); else 0. | integrity-constraint |
| Agent Consistency Score (Agent Con.) ↑ | 1 if the same single agent moves smoothly/consistently across all frames (continuous trajectory; exactly one agent); else 0. | integrity-constraint |
| Spatial Alignment Score (Spa. Ali.) ↑ | 1 if heading, motion direction, and elevation changes stay coherent with the physical layout (initial position identifiable; initial facing aligns with first movement); else 0. | integrity-constraint |
Instruction Following (VLM-judge, binary)
| Metric | Meaning | Role |
|---|---|---|
| Destination Integrity Score (Des. Inte.) ↑ | 1 if the red-marked target stays unchanged (size/position/texture/appearance), no hallucinated alternatives, and the agent stops within it; else 0. | integrity-constraint |
| Scene Consistency Score (Scene Con.) ↑ | 1 if the environment stays static (nothing appears/disappears/deforms/shifts) across all frames; else 0. | integrity-constraint |
Gate (composite) metrics
| Gate | Definition | Applies to |
|---|---|---|
| Success (3D) with Original Destination | S.S. 3D ∧ Des. Inte. ∧ Scene Con. | Last-Mile (T5), 3D R.-W. (T7), SLAG (T8) |
| Success (2D) Original Destination | S.S. 2D ∧ Des. Inte. ∧ Scene Con. | Top-down (T6) |
| Physics Validness / Physical Validity | Obj. Sem. ∧ Agent Con. ∧ Spa. Ali. | all four nav tasks |
Gated primary (Overall Success) — verbatim per task
- T5 Last-Mile & T7 3D R.-W.: "Overall Success = Success Score 3D ∧ Oracle Success Score 3D ∧ Object Semantic ∧ Agent Consistency ∧ Spatial Alignment ∧ Destination Integrity ∧ Scene Consistency; a sample passes only when all seven binary checks are 1."
- T6 Top-down: "Overall Success = Success Score 2D ∧ Oracle Success Score 2D ∧ Object Semantic ∧ Agent Consistency ∧ Spatial Alignment ∧ Destination Integrity ∧ Scene Consistency; a sample passes only when all seven binary checks are 1."
- T8 SLAG: "Overall Success = Success Score 2D ∧ Oracle Success Score 2D ∧ Success Score 3D ∧ Oracle Success Score 3D ∧ Trajectory Alignment ∧ Object Semantic ∧ Agent Consistency ∧ Spatial Alignment ∧ Destination Integrity ∧ Scene Consistency; a sample passes only when all ten binary checks are 1." (Only a 3D destination gate is possible because SLAG's 2D map is generated, not given.)
Shared statement (G.3.4): "a sample is considered correct only if all fine-grained evaluation metrics simultaneously achieve a score of 1." The Overall Success is the Table 4 / Table 26 primary score.
Geo-Align (T5 only) — deterministic geometric metrics
SR (Success Rate), SPL (Success weighted by Path Length), nDTW / sDTW (normalized / success-weighted Dynamic Time Warping), ATE (Absolute Trajectory Error), RPE (Relative Pose Error), Path Length Pred vs GT, Nav (final distance error), Dir (final heading misalignment). GT via manual control in Habitat; predicted poses via VGGT 3D reconstruction. SR/SPL/nDTW/sDTW higher = better; ATE/RPE/Nav/Dir = errors (lower = better).
View-Syn (T6, T7, T8) — deterministic distributional metrics
FVD (video), FID (image), SSIM, PSNR (higher = better), LPIPS (lower = better). Raw FID/FVD normalized into [0,1] by Eq. (1):
S = max( 0, min( 1, 1 − (d − d_min) / (d_max − d_min) ) )
where d ∈ {FID, FVD} and d_min, d_max are the empirical bounds of the metric's distribution. Higher normalized FVD/FID = better. (View-Syn K.3.2 results table for SLAG: not specified in paper — the header exists but no table.)
T9 & T10 — Physical Commonsense (shared metric suite, L.3)
Evaluation is exclusive to video generative models (static image generators cannot represent causal interactions over time). For each generated video the VLM judge (Gemini-2.5-Pro, prompted as a "physics-aware video analysis expert") scores four binary dimensions.
| Metric | Meaning | Method | Values | Role | Gating |
|---|---|---|---|---|---|
| Physics Accuracy ↑ | Whether motion obeys gravity/momentum/friction and interactions are plausible. 0 if objects float, move at unrealistic speeds, wrong trajectories, or deviate from the scenario's Physics Focus. | VLM judge | 0/1 | integrity-constraint | Required = 1 |
| Motion Quality ↑ | Temporal coherence/naturalness; motion follows the expected pattern, smooth and continuous. 0 if jerky, unnatural accelerations, or temporal discontinuities. | VLM judge | 0/1 | reasoning-process | Required = 1 |
| Visual Realism ↑ | Visual fidelity/believability; realistic objects/materials, consistent lighting/shadows, plausible composition. 0 if significant artifacts/glitches or lacks photorealism. | VLM judge | 0/1 | integrity-constraint | Required = 1 |
| Prompt Adherence ↑ | Whether the video semantically matches the input (key objects/setting present; described action occurs). 0 if significant mismatches. | VLM judge | 0/1 | outcome-correctness | Required = 1 |
| Overall Success ↑ | Holistic generation capability. | VLM judge | 0/1 | primary | See below |
Gated primary (verbatim): "A generated video is marked as successful (1) if and only if it satisfies all four fine-grained metrics simultaneously." I.e. Overall = Physics Accuracy ∧ Motion Quality ∧ Visual Realism ∧ Prompt Adherence. All four required.
Gated primary score — summary
Across all tasks the primary/"Overall" score is a strict AND-gate: a sample scores 1 only when every required binary check holds simultaneously; a single failure yields 0. Fractional metrics (Constraint Violation, Completion Accuracy) enter the gate at their strict endpoints (violation = 0, accuracy = perfect/1.0).
| Task | Gated primary formula | # required checks | Non-gating diagnostics |
|---|---|---|---|
| T1 Maze (VLM) | MC=0 ∧ CW=0 ∧ TA=1 | 3 | Action Reflection |
| T1 Maze (pixel) | layout IoU>0.8 (video: min-IoU) ∧ no wall crossing (video: ratio ≤5%) ∧ target achievement>0.95 ∧ no excessive reflection | 4 | — |
| T2 Sudoku | Clues Changed=0 ∧ Constraint Violation=0 ∧ Completion Accuracy=1 | 3 | Action Reflection |
| T3 ARC | Valid Solution = exact match to ground truth | 1 (independent) | Pattern Recognition, Grid Integrity, Color Accuracy (cascade, not a stated gate) |
| T4 Math | FC=1 ∧ IC=1 | 2 | Action Reflection |
| T5 Last-Mile | S.S.3D ∧ O.S.3D ∧ ObjSem ∧ AgentCon ∧ SpaAli ∧ DesInte ∧ SceneCon | 7 | Gate composites (Succ3D-OrigDest, Physics Validness), Geo-Align suite |
| T6 Top-down | S.S.2D ∧ O.S.2D ∧ ObjSem ∧ AgentCon ∧ SpaAli ∧ DesInte ∧ SceneCon | 7 | Gate composites, View-Syn suite |
| T7 3D R.-W. | S.S.3D ∧ O.S.3D ∧ ObjSem ∧ AgentCon ∧ SpaAli ∧ DesInte ∧ SceneCon | 7 | Gate composites, View-Syn suite |
| T8 SLAG | S.S.2D ∧ O.S.2D ∧ S.S.3D ∧ O.S.3D ∧ TrajAlign ∧ ObjSem ∧ AgentCon ∧ SpaAli ∧ DesInte ∧ SceneCon | 10 | Gate composites, View-Syn suite |
| T9 Physical Concepts | Physics Accuracy ∧ Motion Quality ∧ Visual Realism ∧ Prompt Adherence | 4 | — |
| T10 Sports | Physics Accuracy ∧ Motion Quality ∧ Visual Realism ∧ Prompt Adherence | 4 | — |
Deterministic evaluation (non-VLM)
Maze — pixel-based path reconstruction (C.5.3)
"A fully deterministic pixel-based evaluation pipeline that relies solely on classical computer vision operations, without invoking any vision–language or large language models." Color segmentation isolates black walls, green start, red target, blue path. Four metric classes:
- Wall Crossing Detection — checks whether the generated blue path intersects walls, using erosion-based tolerance for anti-aliasing; crossings flagged only when the intersection ratio exceeds a small threshold.
- Layout Matching — structural consistency via IoU between wall regions and centroid distances for start/target markers within a fixed pixel tolerance.
- Path Matching — dilates thin generated paths to cell width and computes IoU, precision, recall, and a cell-level target achievement score.
- Reflection/Exploration Detection — analyzes the skeletonized path to identify loops and excessive detours (backtracking/exploration beyond the optimal solution).
Video extension: frames extracted — 192 frames at 24 fps for Veo-3, 360 frames for Sora-2, 120 frames for Wan-2.2. The green agent marker is tracked across frames to reconstruct the full traversal; the accumulated path is compared against the ground-truth solution image (not intermediate frames). Wall Crossing uses stronger erosion; Layout Matching is computed per frame and the minimum score across all frames is reported; wall crossing flagged only if the crossing ratio exceeds 5%. (Example diagnostics: Veo-3 layout IoU averages 0.6573, GPT-4o-image 0.3210, Sora-2 0.1260; Veo-3 Phi coefficient ϕ = 0.3821 between layout failure and wall crossing.)
Sudoku — OCR-based (D.4.2)
Digit extraction via PaddleOCR with adaptive thresholding, denoising, and cell segmentation (per 4×4 or 9×9 resolution); OCR applied per cell; recognized symbols filtered to valid digits 1–9; reconstructed grid checked against the original puzzle, the ground-truth solution, and standard Sudoku constraints. "Without invoking any vision–language or large language models." Video: the maximum available frames (192 for Veo-3 at 24 fps, 360 for Sora-2, 120 for Wan-2.2) are OCR'd; Constraint Violation and Completion Accuracy are evaluated on the final frame; the Action Reflection thresholds (≤3 avg cell updates/frame for 4×4, ≤5 for 9×9; ≤25% cells changing per frame; ≥20% frames active) verify gradual filling.
VLM judge (Gemini-2.5-Pro)
- Judge model: Gemini-2.5-Pro (Comanici et al., 2025) for all tasks lacking reliable pixel-/symbol-level verification: ARC, the four Embodied Navigation tasks, and Physical Commonsense; it also instantiates the Maze/Sudoku VLM-based pipelines. Exact numeric version/date beyond "2.5-Pro":
not specified in paper. - Inputs: the model-generated video or image, the ground-truth solution/target where one exists, and a structured, task-specific evaluation prompt (with modality-specific variants for video vs. image). For navigation it receives the generation plus task-specific context and returns binary metric scores and brief justifications. For Physical Commonsense it is prompted as a "physics-aware video analysis expert."
- Behaviour: produces the yes/no (or fractional, for Sudoku CV/CA) judgments; binary metrics and the gated Overall are then computed by rule.
- Text-solution baselines: Gemini-3-Flash and Gemini-3-Pro appear only as models under evaluation on Maze, Sudoku, and Math (not as the judge).
- Judge prompt templates: the full verbatim VLM judge/evaluation prompt templates are
not specified in paperfor every task — the paper describes the judge only as "a structured evaluation prompt with task-specific rubrics." (Case-study figures expose example per-metric judge rationales for Physical Commonsense — e.g. Figures 28–30 — but not a reusable template.)
Evaluation reliability (Table 5 — human vs. evaluator agreement)
Three graduate-level students each independently scored balanced subsets (several dozen to several hundred outputs per task); each output judged by ≥2 annotators, disagreements resolved by discussion; the same fine-grained gated rubric as the automatic evaluators; binary 0/1 per criterion; video inspected frame by frame; paid $25/hour. Veo-3 annotated throughout; Nano-banana Pro additionally annotated on the embodied tasks.
| Task (evaluator) | Sora-2 | Veo-3 | Wan-2.2 | Nano-banana Pro |
|---|---|---|---|---|
| Maze (Pixel) | 100.00% | 95.56% | 100.00% | — |
| Sudoku (OCR) | 100.00% | 100.00% | 100.00% | — |
| ARC (VLM) | — | 94.90% | — | — |
| 3D R.-W. Nav. (VLM) | — | 75.33% | — | 21.67% |
| Last-Mile Nav. (VLM) | — | 53.50% | — | 30.00% |
| Top-down Nav. (VLM) | — | 79.15% | — | 51.69% |
| SLAG (VLM) | — | 88.45% | — | 62.39% |
| Physics (VLM) | — | 65.31% | — | — |
How to discount the results. The deterministic checks are highly reliable (Maze pixel 95.56–100%, Sudoku OCR 100%), and ARC's VLM agreement is high (94.90%). Reliability is markedly lower for the VLM-judged Embodied Navigation and Physical Commonsense tasks, and lowest of all on the image model there:
- Last-Mile Nav. has the weakest human–evaluator agreement for the reference video model (Veo-3 53.50%) and only 30.00% for Nano-banana Pro — the auto-metrics also over-rate Veo-3 (73.33% auto vs 25.00% human, per H.3). Treat Last-Mile fine-grained/holistic numbers as the least trustworthy.
- 3D R.-W. Nav. shows only 21.67% agreement on Nano-banana Pro (Veo-3 75.33%) — image-model 3D results should be read with strong caution.
- Top-down Nav. (Veo-3 79.15%, Nano-banana Pro 51.69%) and Physics (Veo-3 65.31%) are intermediate; still substantially below the deterministic tasks.
- SLAG is comparatively better among the VLM tasks (Veo-3 88.45%, Nano-banana Pro 62.39%) but the image-model figure again lags.
In short: Maze, Sudoku, and ARC scores are reliable; Embodied Navigation and Physical Commonsense scores — especially Last-Mile and the image-model 3D numbers — should be discounted, since human annotators frequently disagreed with the automatic evaluator.