Benchmark Taxonomy for Video-VLM Efficiency Research

Date: 2026-04-16 Parent: literature-map-2026-04-16.md

This taxonomy states why each benchmark is used or deferred, what it controls for, and where our current slices stand. A benchmark's presence here does not imply a local result.

Axes we care about

Axis	Question
temporal-bias controls	does the benchmark suppress single-frame / few-frame shortcuts?
task format	multiple choice / open-ended VQA / long-form answer
scorer	exact-match / strict parser / judge-model / metric-based
frame structure	short clip / medium / long / streaming
current local slice	what have we frozen and where?
status	active / deferred with reason

The four benchmarks in scope

TOMATO (arXiv 2410.23266) — primary temporal stress benchmark

• Temporal-bias controls: explicitly designed to force multi-frame gain, order sensitivity, and frame-information disparity. Items are constructed so naive single-frame or out-of-order answers fail.
• Task format: multiple choice (letters A-F).
• Scorer: strict-choice parser on the local runner (exact letter match on the first recognized choice-letter in the generation, with the loose fallback parser kept as a documented backup).
• Frame structure: short clips (~8 uniform frames on our stack).
• Current local slices:
  • tomato_motion_dev_v1.toml (15 items, groups: direction, rotation, shape_trend) — used for planner search
  • tomato_motion_holdout_v1.toml (15 disjoint items, same groups) — phase 1.12 holdout
• Why we use it: our current strongest motion-reasoning test, and the benchmark where the imported target claims near-perfect agreement.
• Status: active, primary temporal-reasoning benchmark.
• Current release role: Phase 1.20 N=30 enlargement landed and supplies the clean TOMATO routing tie against dense-8 at lower effective budget.

MVBench (arXiv 2311.17005) — broader video-reasoning benchmark

• Temporal-bias controls: the benchmark paper claims no task should be solvable with a single frame. In practice, our chosen motion-heavy subset retains partial first-frame solvability on some items (noted in phase 1.47 ablation). Treat as broader video-reasoning, not pure temporal stress.
• Task format: multiple choice.
• Scorer: exact-letter match.
• Frame structure: short clips (hosted predecessor-style selection, 8 uniform frames on our stack).
• Current local slices:
  • mvbench_motion_dev_v1.toml (15 items, groups: action_localization, fine_grained_action, object_interaction, moving_direction, moving_attribute) — phase 1.11 planner grid
  • mvbench_motion_holdout_v1.toml (15 disjoint items, same groups) — phase 1.12 holdout
• Why we use it: validates cross-benchmark generalization and gives us a different content mix than TOMATO.
• Status: active but narrower than the paper's scope. Do not claim our MVBench slice is representative of "MVBench" as a whole.
• Current release role: Phase 1.21 N=30 enlargement landed and supplies the clean MVBench base-policy comparison against dense-6. The sticky4 refinement remains dirty-tree supplementary until rerun clean.

TempCompass (arXiv 2403.00476) — temporal-aspect-isolation benchmark

• Temporal-bias controls: explicitly uses "conflicting videos with the same static content but different temporal aspects" to suppress single-frame bias. Aspects directly isolated: speed, direction, event order, attribute change.
• Task format: mix of multiple choice and open-ended VQA.
• Scorer: task-specific (judge model for open-ended).
• Frame structure: short clips.
• Why we want it: TempCompass's weak categories (direction, speed, order, attribute change) are exactly the ones showing failure patterns on our current MVBench + TOMATO work. It is a better next diagnosis benchmark than a broader one.
• Status: preregistered as phase 1.25; not yet ingested. Corpus + parser work needed.
• Priority: should move up in the queue per the 2026-04-16 audit; defer to after the sticky-dynamic + projector-group + N=30 work only because those are in-flight.

Video-MME (arXiv 2405.21075) — long-horizon general-video benchmark

• Temporal-bias controls: general video QA; not specifically a temporal-stress benchmark. Multi-modal (includes audio / long clips / multiple camera angles).
• Task format: multiple choice.
• Scorer: exact-match.
• Frame structure: short / medium / long / very long.
• Why we use it: broader external-validity check after the TOMATO and MVBench mechanism slices; also the paper-facing benchmark for frame-count and C-VISION denominator evidence.
• Status: active / evaluated. Phase 1.41 earned the Qwen VideoMME breadth gate at 8f and strengthened it at 16f/32f; disjoint 16f holdout landed later. The dev-split 16f long-bucket regression did not replicate on holdout, so VideoMME supports breadth and denominator accounting, not a universal frame-scaling law. Do not use it as the primary diagnosis benchmark; the task heterogeneity dilutes failure-mode signal.

EgoSchema (arXiv 2308.09126) — low-reuse long egocentric robustness

• Temporal-bias controls: 3-minute egocentric clips from Ego4D. Constant camera motion (head, hands) means most frames are NOVEL — the worst case for temporal caching. Functions as the stress test for the opposite end of the reuse spectrum.
• Task format: multiple choice.
• Scorer: exact-letter match.
• Frame structure: long (3-minute clips, 32 frames sampled).
• Why we want it: pre-release source WP-2.7.1 reports Qwen2.5-VL-7B achieves 100% byte-identical output at 29.9% avg reuse (range 20.1%–65.1%) with 0.0% accuracy delta. If our planner sustains this low-reuse robustness, claim #4 (budget-placement-over-quantity) gains stronger evidence: caching value is not "only for high-reuse content."
• Status: preregistered as phase 1.43 (post-1.41 VideoMME).
• Priority: medium — executes after VideoMME lane.

Not in scope (but occasionally referenced)

• UCF-Crime streaming anomaly: what CodecSight evaluates. Different task format (video-level F1), different failure mode. Only referenced for Track B streaming-window harness comparison (phase 1.30), not for Track A quality claims.
• PerceptionTest, NextQA, ActivityNet-QA, CVRR-ES, etc.: used by CoPE-VideoLM as their benchmark suite. Not aligned to our temporal-reasoning focus; not planned.
• VideoMME: occasionally cited as the CoPE / FastVID reference; treat as external validity not diagnosis.

What this taxonomy means for paper framing

1. Our Track A story must survive on TOMATO (primary) and MVBench (secondary) with N=30. TempCompass becomes the third benchmark when budget permits, and its per-aspect breakdown is the most useful mechanistic evidence for "where budget placement matters."
2. Do not generalize our MVBench motion slice to "MVBench" — be explicit about the motion-heavy group subset.
3. VideoMME is evaluated as a breadth and denominator-accounting check, not as the primary diagnosis or method-comparison arena.