GeoQA

Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing, Liang Lin. “GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning”. Annual Meeting of the Association for Computational Linguistics (Findings of ACL 2021). [PDF]

Download GeoQA benchmark: [Google Drive]

01/10/2022 Update: We find some minor errors in data annotation. These errors have been fixed, please download the latest GeoQA benchmark. Besides, both arXiv paper and this project have also been updated.

Environment

python=3.6

allennlp==0.9.0

Document for allennlp

Usage of NGS-Auxiliary

Preparing

git clone https://github.com/chen-judge/GeoQA.git

cd GeoQA

pip install -r requirements.txt

Download the data.zip, move it to GeoQA path, and unzip it.

Training

allennlp train config/NGS_Aux.json --include-package NGS_Aux -s save/test

Evaluation

Evaluate your trained model:

allennlp evaluate save/test  data/GeoQA3/test.pk --include-package NGS_Aux_test --cuda-device 0

Or, you can use our checkpoint NGS_Aux_CKPT.zip, move it to save path, unzip it, and run:

allennlp evaluate save/NGS_Aux_CKPT  data/GeoQA3/test.pk --include-package NGS_Aux_test --cuda-device 0

The result of our checkpoint should be:

Method	Acc	Angle	Length	Other
NGS-Auxiliary	60.0	71.5	48.8	29.6