RpBERT
March 31, 2023 ยท View on GitHub
This is a implementation of the paper RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER.
Requirements
Datasets
- Download multi-modal NER dataset Twitter-15 (Zhang et al., 2018) from here to this path.
- Download multi-modal NER dataset Twitter-17 (Lu et al., 2018) to this path.
- Download text-image relationship dataset (Vempala et al., 2019) from here to this path.
Run loader.py to make sure the statistics is identical as (Zhang et al., 2018) and (Lu et al., 2018).
| Twitter-15 | NUM | PER | LOC | ORG | MISC |
|---|---|---|---|---|---|
| Training | 4000 | 2217 | 2091 | 928 | 940 |
| Development | 1000 | 552 | 522 | 247 | 225 |
| Testing | 3257 | 1816 | 1697 | 839 | 726 |
| Twitter-17 | NUM | TOKEN |
|---|---|---|
| Training | 4290 | 68655 |
| Development | 1432 | 22872 |
| Testing | 1459 | 23051 |
Models
- Download pre-trained ResNet-101 weights from here to this path.
- Download pre-trained BERT-Base weights from here to this path.
- Download pre-trained word embeddings from here to this path.
Libraries
- tqdm
- Pillow
- numpy
- torch
- torchvision
- transformers
- flair
- pytorch-crf
Usage
# BERT-BiLSTM-CRF
python main.py --stacked --rnn --crf --dataset [dataset_id] --cuda [gpu_id]
# RpBERT-BiLSTM-CRF
python main.py --stacked --rnn --crf --encoder_v resnet101 --aux --gate --dataset [dataset_id] --cuda [gpu_id]