VGEBench

Code and data for Towards Generalizable Visually Grounded Exploration of Household Devices.

Contents

• bench_close.py: runs the closed/API-based VLM benchmark.
• bench_open.py: runs a local open-source VLM benchmark.
• evaluation.py: evaluates benchmark result files and reports SR, SSR, SPL, State-F1, EIR, TIR, VSPS, and EER.
• polished_QA.json: the task file used by bench_close.py.
• state_machine.json: universal category-level state machines.
• models/api_model_infer.py: OpenAI-compatible API inference wrapper.
• models/local_model_infer.py: local VLM inference wrapper.
• models/model_api_key.txt: API key template. Fill in your API key in this file.
• prompts/system prompt.txt: benchmark system prompt.
• image_view/: placeholder for the visual assets and per-device state machines.

Setup

Install the Python dependencies:

pip install openai pandas openpyxl pillow numpy

For local VLM evaluation, also install compatible PyTorch and Transformers packages:

pip install torch transformers torchvision

Download the image_view assets separately and extract them into the repository root. The image_view data can be downloaded from the VGEBench Hugging Face dataset repository:

https://huggingface.co/datasets/codseeker/VGEBench

After downloading, place the extracted image_view/ directory in the repository root.

Configure your API key in models/model_api_key.txt:

openAI: sk-your-api-key

Run the benchmark

Run the default GPT-5-mini configuration from the repository root:

python bench_close.py

The default output is written to results/result_gpt_5_mini.xlsx. Use --output-file to choose a different result filename:

python bench_close.py --output-file my_result.xlsx

The benchmark saves progress after each episode and resumes from an existing output file by id.

Evaluate results

Evaluate a specific result file by passing its filename, without the results/ prefix:

python evaluation.py result_gpt_5_mini.xlsx

Detailed per-episode metrics are saved under results/evaluation_scores/.

Run a local open-source model

The default local model is Qwen3_VL_8B. Set VGEBENCH_MODEL_PATH when the model is stored at a local path:

VGEBENCH_MODEL_PATH=/path/to/Qwen3-VL-8B-Instruct python bench_open.py --model Qwen3_VL_8B

Dataset statistics

VGEBench contains 26 device categories, 968 device instances, 7,888 views, 3,712 unique interactive components, 7,264 annotated bounding boxes, 6,352 SSM states, 15,861 SSM transitions, and 14,953 task episodes.

Citation

Feel free to cite this work if VGEBench is useful to you.

@inproceedings{zheng2026vgebench,
  title={Towards Generalizable Visually Grounded Exploration of Household Devices},
  author={Linhao Zheng and Zeming Liu and Wangke Chen and Li Zeng and Wanxiang Che and Heyan Huang and Yuhang Guo},
  booktitle={Findings of the Association for Computational Linguistics: EMNLP 2026},
  year={2026}
}