Evaluation

June 28, 2024 ยท View on GitHub

To ensure the reproducibility, we evaluate the models with greedy decoding. We do not evaluate using beam search to make the inference process consistent with the chat demo of real-time outputs.

Currently, we mostly utilize the official toolkit or server for the evaluation.

Evaluate on Custom Datasets

You can evaluate MoVA on your custom datasets by converting your dataset to MoVA's jsonl format, and evaluate using model_vqa.py.

Below we provide a general guideline for evaluating datasets with some common formats.

  1. Short-answer (e.g. VQAv2, MME).
<question>
Answer the question using a single word or phrase.
  1. Option-only for multiple-choice (e.g. MMBench, SEED-Bench).
<question>
A. <option_1>
B. <option_2>
C. <option_3>
D. <option_4>
Answer with the option's letter from the given choices directly.
  1. Natural QA (e.g. MathVista, MathVerse).

No postprocessing is needed.

Scripts

We use the custom annotations of LLaVA, please first download eval.zip and put the annotations into ./playground/data/eval. Besides, the annotations of other benchmarks (e.g., DocVQA) are already available in ./playground/data/eval.

VQAv2

  1. Download test2015 and put it under ./playground/data/eval/vqav2.
  2. Multi-GPU inference.
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/mova/eval/vqav2.sh
  1. Submit the results to the evaluation server: ./playground/data/eval/vqav2/answers_upload.

GQA

  1. Download the data and evaluation scripts following the official instructions and put under ./playground/data/eval/gqa/data. You may need to modify eval.py as this due to the missing assets in the GQA v1.2 release.
  2. Multi-GPU inference.
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/mova/eval/gqa.sh

ScienceQA

  1. Under ./playground/data/eval/scienceqa, download images, pid_splits.json, problems.json from the data/scienceqa folder of the ScienceQA repo.
  2. Single-GPU inference and evaluate.
CUDA_VISIBLE_DEVICES=0 bash scripts/mova/eval/sqa.sh

POPE

  1. Download coco from POPE and put under ./playground/data/eval/pope.
  2. Single-GPU inference and evaluate.
CUDA_VISIBLE_DEVICES=0 bash scripts/mova/eval/pope.sh

MME

  1. Download the data following the official instructions here.
  2. Downloaded images to MME_Benchmark_release_version.
  3. put the official eval_tool and MME_Benchmark_release_version under ./playground/data/eval/MME.
  4. Single-GPU inference and evaluate.
CUDA_VISIBLE_DEVICES=0 bash scripts/mova/eval/mme.sh

MMBench

  1. Download mmbench_dev_20230712.tsv and put under ./playground/data/eval/mmbench.
  2. Single-GPU inference.
CUDA_VISIBLE_DEVICES=0 bash scripts/mova/eval/mmbench.sh
  1. Submit the results to the evaluation server: ./playground/data/eval/mmbench/answers_upload/mmbench_dev_20230712.

MMBench-CN

  1. Download mmbench_dev_cn_20231003.tsv and put under ./playground/data/eval/mmbench.
  2. Single-GPU inference.
CUDA_VISIBLE_DEVICES=0 bash scripts/mova/eval/mmbench_cn.sh
  1. Submit the results to the evaluation server: ./playground/data/eval/mmbench/answers_upload/mmbench_dev_cn_20231003.

TextVQA

  1. Download TextVQA_0.5.1_val.json and images and extract to ./playground/data/eval/textvqa.
  2. Single-GPU inference and evaluate.
CUDA_VISIBLE_DEVICES=0 bash scripts/mova/eval/textvqa.sh

DocVQA

  1. Download val images and test images and extract to ./playground/data/eval/docvqa.
  2. Single-GPU inference and evaluate.
CUDA_VISIBLE_DEVICES=0 bash scripts/mova/eval/docvqa_val.sh
CUDA_VISIBLE_DEVICES=0 bash scripts/mova/eval/docvqa_test.sh
  1. Submit the test results to the evaluation server.

ChartQA

  1. Download images and extract to ./playground/data/eval/chartqa.
  2. Single-GPU inference and evaluate.
CUDA_VISIBLE_DEVICES=0 bash scripts/mova/eval/chartqa.sh

AI2D

  1. Download images and extract to ./playground/data/eval/ai2d.
  2. Single-GPU inference and evaluate.
CUDA_VISIBLE_DEVICES=0 bash scripts/mova/eval/ai2d.sh

Q-Bench

  1. Download llvisionqa_dev.json (for dev-subset) and llvisionqa_test.json (for test-subset). Put them under ./playground/data/eval/qbench.
  2. Download and extract images and put all the images directly under ./playground/data/eval/qbench/images_llviqionqa.
  3. Single-GPU inference (change dev to test for evaluation on test set).
CUDA_VISIBLE_DEVICES=0 bash scripts/mova/eval/qbench.sh dev
  1. Submit the results by instruction here: ./playground/data/eval/qbench/llvisionqa_dev_answers.jsonl.