📖 Overview

Existing spatial benchmarks usually require coordinates, option labels, or textual descriptions. This creates an answer-interface mismatch for image-generation models: they can express spatial judgments by pointing, marking, masking, or drawing in pixel space, but those visual answers fall outside the original evaluator.

ProVisE changes only the response interface. A task-aware router assigns a visual protocol whose guidance prompt and parser are fixed before generation. The model produces a protocol-constrained visual answer, the parser converts it into the required structured prediction, and the original benchmark metric scores the result. Text-output VLMs continue to answer in the original answer space, enabling comparison under shared task semantics and metrics.

benchmark task
  -> task-aware protocol routing
  -> protocol-constrained visual response
  -> structured prediction
  -> original benchmark metric

Building on ProVisE, SpatialGen-Bench contains 470 curated samples across 14 subtasks and four capability levels: perception, understanding, reasoning, and interaction. The study finds complementary strengths: image-generation models are competitive when judgments can be externalized in pixels, while text-output VLMs remain stronger in complex spatial reasoning. ProVisE further automates adaptation to new benchmarks through protocol reuse, construction from Parser Ops, and an explicitly labeled fallback when deterministic readout is unavailable.

🧪 Supported Benchmarks

Each validated benchmark has an independent artifact in the Protocol Pool.

Benchmark	Tasks	Protocol routes	Official resource	Protocol
SpatialGen-Bench	14	Manual 11 + Agentic 3	Hugging Face	View
EmbSpatial-Bench	1	Build 1	GitHub	View
OmniSpatial	5	Fallback 5	Hugging Face	View
Q-Spatial+	3	Build 3	Hugging Face	View
RoboSpatial-Home	3	Build 2 + Reuse 1	Hugging Face	View
SAT	8	Build 5 + Fallback 3	GitHub	View
RoboAfford	3	Reuse 3	GitHub	View

🔄 Continuously updated with newly validated spatial benchmarks and protocol artifacts.

🚀 Quick Start

1. Install

git clone https://github.com/ZJU-OmniAI/ProVisE.git
cd ProVisE

conda create -n provise python=3.10 -y
conda activate provise
pip install -e ".[agentic]"

2. Configure OpenAI

Set the API key in your shell or place the same variable in a local .env file:

export OPENAI_API_KEY="your_openai_api_key"

Protocol construction defaults to gpt-5.4 for the agent and parser, with gpt-image-2 for visual smoke validation.

3. Run SpatialGen-Bench

hf download wx91726/SpatialGen-Bench \
  --repo-type dataset \
  --local-dir benchmarks/SpatialGen-Bench

provise run \
  --source benchmarks/SpatialGen-Bench \
  --model gpt-image-2

The first command downloads the public 470-sample benchmark from Hugging Face. ProVisE then normalizes the package, constructs and validates task protocols, and evaluates a bounded pilot. Add --full after the pilot succeeds. For another benchmark, replace --source with its downloaded directory or a normalized ProVisE package.

🗃️ Protocol Pool

For a fair model comparison, build one versioned protocol artifact and reuse it across every evaluated model:

provise build --source /path/to/benchmark

provise evaluate \
  --protocol outputs/agentic_runs/<benchmark>/<run> \
  --model gpt-image-2 \
  --full

The build command prints the protocol directory consumed by provise evaluate. To process a directory of downloaded benchmarks, run the suite command:

provise suite \
  --benchmark-root /path/to/downloaded/benchmarks \
  --model gpt-image-2

Each benchmark receives its own protocol directory; missing datasets are reported separately and are never counted as model failures. Audited artifacts live in configs/protocols:

configs/protocols/<benchmark>/
  configs/       benchmark routes, prompts, readout, and metric contract
  generated/     task-specific protocol definitions

Benchmark media is not redistributed. To evaluate a published protocol, place or link its normalized package under benchmark/; see the Protocol Pool documentation.

🧩 Extending ProVisE

🤝 Issues and pull requests are welcome. Use issues for bug reports, benchmark requests, and proposed public interfaces; submit focused PRs for benchmark adapters, visual protocols, or Parser Ops.

Contribution	Location	Required contract
Benchmark adapter	benchmark.yaml, data.jsonl, and assets/	Task/media mapping, answer schema, metric contract, and a smoke fixture
Visual protocol	configs/protocol_specs	Benchmark-independent response contract, parser, output kind, compatible metrics, and focused tests
Parser Op	provise/parser_ops	Typed inputs, parameters, deterministic behavior, and unit tests

Start with the Unified Sample Schema and Agentic Benchmark Adapter. Before submitting a PR, install the contributor dependencies and run the repository checks:

pip install -e ".[agentic,dev]"
ruff check .
pytest -q

🏗️ Repository Layout

ProVisE/
├── assets/                         README figures and branding
├── configs/
│   ├── benchmark_suites/           Validated multi-benchmark run definition
│   ├── protocol_specs/             Global protocol catalog and runtime adapters
│   └── protocols/                  Published benchmark protocol artifacts
├── docs/                           Schemas and benchmark-adapter guides
├── provise/
│   ├── benchmark/                  Ingestion, validation, and sample contracts
│   ├── commands/                   CLI workflow implementations
│   ├── evaluation/                 Evaluation runtime, metrics, and summaries
│   ├── models/                     Image-generation and VLM adapters
│   ├── parser_ops/                 Typed visual readout operators
│   ├── protocol_agent/             Protocol planning and contract compilation
│   ├── protocols/                  Executable protocols and registry
│   ├── cli.py                      `provise` command-line entry point
│   └── reporting.py                Terminal progress and status reporting
├── scripts/                        Conversion and low-level utility scripts
└── tests/                          Unit, integration, and smoke tests
    └── fixtures/                   Minimal benchmark packages for testing

See the configuration layout for the ownership and output rules of each directory. The installed provise command is the supported user interface. Files under scripts/ are retained only for benchmark conversion and low-level smoke or evaluation workflows. Downloaded benchmarks and runtime outputs are intentionally not tracked.

🙏 Acknowledgements

We sincerely appreciate CountBench, BLINK, EgoOrientBench, VSR, ViewSpatial-Bench, MindCube, VisWorld-Eval, RoboAfford-Eval, ShareRobot-Bench, PhysBench, SPHERE-VLM, and RefCOCOg for their public datasets, task designs, and evaluation resources.

📝 Citation

@article{wang2026showdonttell,
  title  = {Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text},
  author = {Wang, Xu and Yao, Kaixiang and Pan, Miao and Zhou, Xiaohe and Liu, Xuanyu and Zhang, Wenqi and Zhang, Xuhong},
  journal = {arXiv preprint arXiv:2607.21072},
  year   = {2026}
}

⚖️ License

ProVisE source code is released under the Apache License 2.0. Third-party benchmarks, models, and assets remain subject to their original licenses and terms.