ToolVQA: A Dataset for Real-World VQA with External Tools (ICCV 2025)

November 3, 2025 · View on GitHub

Authors: Shaofeng Yin, Ting Lei, Yang Liu

1. Introduction 📣

Integrating external tools into Large Foundation Models (LFMs) has emerged as a promising approach to enhance their problem-solving capabilities. While existing studies have demonstrated strong performance in tool-augmented Visual Question Answering (VQA), recent benchmarks reveal significant gaps in real-world tool-use proficiency, particularly in functionally diverse multimodal settings requiring multi-step reasoning.

In this work, we introduce ToolVQA, a large-scale multimodal dataset comprising 23K samples, designed to bridge this gap. Unlike previous datasets that rely on synthetic scenarios and simplified queries, ToolVQA features real-world visual contexts and challenging implicit multi-step reasoning tasks, better aligning with real user interactions.

To construct this dataset, we propose ToolEngine, a novel data generation pipeline that employs image-guided Depth-First Search (DFS) with a Longest Common Subsequence (LCS)-based example matching mechanism to simulate human-like tool-use reasoning. ToolVQA encompasses 10 multimodal tools across 7 diverse domains, with an average inference length of 2.78 reasoning steps per sample.

The LLaVA-7B model fine-tuned on ToolVQA not only achieves impressive performance on the ToolVQA test set, but also surpasses the large closed-source model GPT-3.5-turbo on five out-of-distribution (OOD) datasets, showing strong generalizability in real-world tool-use scenarios.

The fine-tuned 7B LFMs on ToolVQA not only achieve impressive performance on our test set but also surpass the large close-sourced model GPT-3.5-turbo on various out-of-distribution (OOD) datasets, demonstrating strong generalizability to real-world tool-use scenarios.

For more details, please check the original paper.

2. Implemention 🚀

2.1 Environments

We utilize four open source frameworks to complete our code:

  1. Lmdeploy: deploy LFM locally for training or testing.
  2. Agentlego: deploy external tools and call them through API.
  3. Opencompass: combine lmdeploy and agentlego to evaluate LFM.
  4. Xtuner: fine-tune LFM through well-defined configs.

We highly recommend you to use four different virtual environments to deploy these frameworks separately, rather than installing all of them in one environment. We have exported the deployed environments under the requirements/ directory, and you can directly pip install them.

Our implementation also refers to GTA, and we would like to pay high tribute to all of these great open source work!

2.2 Dataset

You can download the dataset from Google Drive.

There are three files in our dataset:

1. toolvqa_imgs.zip:

23655(21105+2550) images, each image corresponds to an instance of ToolVQA. Please put these images under directory datasets/ToolVQA/.

2. raw_train_21105.json:

21105 training instances. For training, please:

  1. put the data under xtuner/data/train/.
  2. refer to xtuner/data/train/example.json to format the data.
  3. modify the file name in xtuner/llava.py to your file name.

3. raw_test_2550.json:

2550 test instances. For test, please:

  1. refer to opencompass/data/ToolVQA/dataset.json to format the data.
  2. put the data under opencompass/data/ToolVQA/dataset.json, do NOT change the file name.

You should have dataset.json and toolmeta.json under opencompass/data/ToolVQA/ for test.

Each instance is in the format of:

{
    "image_path": "datasets/ToolVQA/2412880.jpg",
    "context": [
        {
            "name": "ImageDescription", # tool name, listed in file 'agentlego/benchmark_toollist.txt'
            "thought": "To solve the problem, we begin by getting a general understanding of the image showing animals used for a specific purpose—in this case, draft horses pulling a plow.", # thought of use this tool in solving process
            "thought_choose": "There is only one choice, so we will directly use it.", # thought of choose this tool in data generating process 
            "input": {
                "image": "datasets/ToolVQA/2412880.jpg"
            }, # tool input
            "output": "The image shows a person riding on a plow being pulled by two draft horses through a field. The person is operating farming equipment, likely for plowing or preparing the land. The scene is set in an outdoor, rural environment, with trees in the background.", # tool output
            "thought_query": "Since we don't know the information in the picture, we first call tool ImageDescription to describe the picture.", # thought of use this tool+input in data generating process  
            "is_important": "yes" # [test set only] human annotated: whether this tool is necessary for solving process (used to calculate Redundancy rate in paper)
        },
        ... # other tool-use steps
    ],
    "question": "What is a common breed of these animals used historically for plowing?", # query
    "ori_question": "What is a common breed of the draft horses in this image used historically for plowing?", # query before gpt-based refinement
    "thought_rethink": "The noun 'draft horses' appears in the ImageDescription. I will replace 'draft horses' with 'these animals' since the question is asking about the specific horses shown in the image.\n\n", # thought of gpt-based refinement
    "thought_question": "The last tool used is the GoogleSearch, which indicates that Percherons are a common breed of draft horses historically used for plowing. The image shows two draft horses being used for plowing, and instead of directly asking about the breed, I should reference the activity in the image. I will avoid mentioning the breed of the horses explicitly and frame the question based on the image description to ensure it can be answered by the information provided by the tool call.", # thought of query
    "answer": "Percherons", # answer
    "type": "multi_obj_no_text", # instance type [multi_obj_no_text / single_obj_no_text / obj_text / no_obj_text] 
    "better_ques": "2", # [test set only] human annotated: whether refined query better than original query (1: yes, 2: no, 3: equal)
    "only_answer": "1", # [test set only] human annotated: whether better query have only one answer (1: yes, 0: no)
    "correct_answer": "yes" # [test set only] human annotated: whether the answer of better query is correct
},

2.3 Data Construction

This part requires agentlego environment.

1. Deploy the tools.

conda activate agentlego
export OPENAI_API_KEY='your_openai_api_key'
export SERPER_API_KEY='your_serper_key_for_google_search_tool' # API for Google Search, register at 'https://serper.dev'
export MATHPIX_APP_ID="your_mathpix_api_id_for_mathocr_tool" # if not use, just delete this line and this tool in 'toollist.txt'.  
export MATHPIX_APP_KEY="your_mathpix_api_key_for_mathocr_tool" # if not use, just delete this line and this tool in 'toollist.txt'. 
cd agentlego
agentlego-server start --port 16181 --extra ./custom_tools.py  `cat toollist.txt` --host 0.0.0.0

2. Preprocess your data.

Process your custom image and its tag (choose from [multi_obj_no_text / single_obj_no_text / obj_text / no_obj_text]) into a dictionary list. In the format of:

[
    {
        "image": "your_file_path.jpg",
        "tag": "multi_obj_no_text"
    },
    ...
]

Please put your data at agentlego/datasets/datas.json.

3. Generate your data.

(1) If you want to generate VQAs for a specific toolchain, run:

cd agentlego/annotators
python build_fix.py

Your can change the specific toolchain in line 39 of build_fix.py. In addition, the code in this file is multi-threaded concurrent. If you want to use the single-threaded version, please refer to the function build_epoch_fixed in build_epoch.py and call it directly.

(2) If you want to generate VQAs for arbitrary toolchains, run:

cd agentlego/annotators
python build_any.py

2.4 Fine-tuning

This section requires xtuner environment.

(1) Use us2gta.py to transfer raw_train_21105.json into GTA format data (To evaluate in opencompass).

(2) Use gta2xtuner.py to transfer GTA format data into xtuner format data (To train in xtuner).

(3) Run

NPROC_PER_NODE=4 xtuner train llava.py --deepspeed deepspeed_zero2

If your environment does not support deepspeed, just remove --deepspeed deepspeed_zero2.

(4) When you get the output checkpoint, you should transfer it into the huggingface format.

To do this, run

cd xtuner

xtuner convert pth_to_hf xtuner/work_dirs/llava_gta/2025xxxx_xxxxxx/vis_data/config.py xtuner/work_dirs/llava_gta/iter_9600.pth xtuner/work_dirs/llava_gta/iter_9600_hf --safe-serialization

xtuner convert merge path/to/models--meta-llama--Meta-Llama-3-8B-Instruct path/to/xtuner_model/llm_adapter path/to/llm_merge --safe-serialization

python convert_xtuner_weights_to_hf.py --text_model_id path/to/xtuner_model/llm_merge --vision_model_id path/to/models--openai--clip-vit-large-patch14-336 --projector_weight path/to/xtuner_model/projector/model.safetensors --save_path path/to/xtuner_model/llava_finetune

You can also download a fine-tuned checkpoint of LLaVA here.

In the following section, you should change path/to/models to your fine-tuned model path.

(5) For any question, please refer to https://xtuner.readthedocs.io/zh-cn/latest/training/multi_modal_dataset.html.

Possible issues:

  1. The fine-tuned model cannot be converted to huggingface format?
  2. The merged model cannot be started?

2.5 Evaluation

This section requires opencompass, lmdeploy, and agentlego environments.

1. Deploy the tools.

cd agentlego

source setup.sh 
# This step includes an export proxy=8890 operation. If the corresponding clash is not opened, an error will be reported.

agentlego-server start --port 16181 --extra ./custom_tools.py  `cat toollist.txt` --host 0.0.0.0 
# --port can specify any port, but the corresponding port must be changed in eval_config later.
# The deployment of this api will occupy about 2G memory, but for unknown reasons, the memory usage will gradually increase during testing, so it is best to reserve about 6G.

2. Start local LFMs' api.

Note: If you are testing api-based LLM, you can skip this step.

conda activate lmdeploy

lmdeploy serve api_server path/to/models --server-port 12580 --model-name your_model_name
# This port can also be specified arbitrarily, and also needs to be changed in eval_config.
# A model basically takes up an entire card's 24G memory.

3. Run evaluations.

cd opencompass
conda activate opencompass

# evaluate from scratch:
python run.py path/to/your_eval_config -p llmit -q auto --max-num-workers 32 --debug --eval_mode=your_eval_mode
# args guideline: opencompass/configs/readme.md
# default output dir: opencompass/outputs/default

# If the test above is interrupted halfway, you can add --reuse 2025xxxx_xxxxxx (the file name of the test output, you can also rename it yourself) to the test command and continue the test.
python run.py path/to/your_eval_config -p llmit -q auto --max-num-workers 32 --debug --eval_mode=your_eval_mode --reuse 2025xxxx_xxxxxx

# want to run the metric again:
python run.py path/to/your_eval_config --max-num-workers 32 --debug --reuse 2025xxxx_xxxxxx --mode eval --eval_mode=your_eval_mode

3. Bibtex 📝

@misc{yin2025toolvqadatasetmultistepreasoning,
  title={ToolVQA: A Dataset for Multi-step Reasoning VQA with External Tools}, 
  author={Shaofeng Yin and Ting Lei and Yang Liu},
  year={2025},
  eprint={2508.03284},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2508.03284}, 
}