JointBERT-paddle

(Unofficial) Paddle implementation of JointBERT: BERT for Joint Intent Classification and Slot Filling

Model Architecture

• Predict intent and slot at the same time from one BERT model (=Joint model)
• total_loss = intent_loss + coef * slot_loss (Change coef with --slot_loss_coef option)
• If you want to use CRF layer, give --use_crf option

Dependencies

• python>=3.6
• paddle == 2.1.3
• paddlenlp == 2.0.0

Dataset

	Train	Dev	Test	Intent Labels	Slot Labels
ATIS	4,478	500	893	21	120
Snips	13,084	700	700	7	72

• The number of labels are based on the train dataset.
• Add UNK for labels (For intent and slot labels which are only shown in dev and test dataset)
• Add PAD for slot label

Training & Evaluation

Style one

• Gitting clone the repo and using the following scripts:

If you want to evaluate our models directly, you can download our trained model parameters from Google Cloud, and change --model_name_or_path in evaluating scripts.

# Train
$ python main.py --task {task_name} \
                  --model_type {model_type} \
                  --model_dir {model_dir_name} \
                  --do_train --do_eval \
                  --use_crf \
                  --model_name_or_path {model_name in the training stage}

# Train For ATIS
$ python main.py --task atis \
                --model_type bert \
                --model_dir atis_model \
                --do_train --do_eval \
                --use_crf \
                --data_dir ../data \
                --model_name_or_path bert-base-uncased \
                --num_train_epochs 30.0
# Train For Snips
$ python main.py --task snips \
                --model_type bert \
                --model_dir snips_model \
                --do_train --do_eval \
                --data_dir ../data \
                --model_name_or_path bert-base-uncased \
                --num_train_epochs 30.0
              
              
# Evaluation For ATIS
$ python main.py --task atis \
                --model_type bert \
                --model_dir atis_model \
                --do_eval \
                --use_crf \
                --data_dir ../data \
                --model_name_or_path bert-base-uncased \
                --num_train_epochs 30.0

# Evaluation For Snips
$ python main.py --task snips \
                --model_type bert \
                --model_dir snips_model \
                --do_eval \
                --data_dir ../data \
                --model_name_or_path bert-base-uncased \
                --num_train_epochs 30.0
                
# Example for evaluation
$ python main.py --task snips \
                --model_type bert \
                --model_dir trained_models/snips_model_nocrf \
                --do_eval \
                --data_dir ../data \
                --model_name_or_path  bert-base-uncased \
                
                

Style two

• Using BaiDu AI Studio and choosing JoingBERT-paddle edition to run our program in here.

Results

		Intent acc (%)	Slot F1 (%)	Sentence acc (%)
Snips	BERT (paper)	98.6	97.0	92.8
	BERT (pytorch)	98.7	96.1	91.1
	BERT (paddle)	98.6	96.1	91.4
	BERT + CRF (paper)	98.4	96.7	92.6
	BERT + CRF (pytorch)	98.6	96.8	93.1
	BERT + CRF (paddle)	98.5	96.8	92.7
ATIS	BERT (paper)	97.5	96.1	88.2
	BERT (pytorch)	97.5	95.9	88.2
	BERT (paddle)	97.5	95.6	87.6
	BERT + CRF (paper)	97.9	96.0	88.6
	BERT + CRF (pytorch)	97.6	95.9	88.7
	BERT + CRF (paddle)	97.4	95.8	88.0