results.md

September 17, 2022 · View on GitHub

Multimodal intent benchmark results on multi-class and binary classification

Methods	ACC_20	F1_20	P_20	R_20	ACC_2	F1_2	P_2	R_2
Human	85.51	85.07	86.37	85.74	94.72	94.67	94.74	94.82
Text-classifier	70.88	67.40	68.07	67.44	88.09	87.96	87.95	88.09
MAG-BERT	72.65	68.64	69.08	69.28	89.24	89.10	89.10	89.13
MulT	72.52	69.25	70.25	69.24	89.19	89.07	89.02	89.18
MISA	72.29	69.32	70.85	69.24	89.21	89.06	89.12	89.06

Results (macro F1-score) of each fine-grained classes in "Express emotions and attitudes"

Methods	Complain	Praise	Apologise	Thank	Criticize	Care	Agree	Taunt	Flaunt	Oppose	Joke
Human	80.08	93.44	96.15	96.90	72.21	96.09	87.21	65.55	78.10	69.04	72.22
Text-classifier	64.36	85.69	97.93	97.22	47.06	87.42	94.26	15.53	46.12	32.32	27.42
MAG-BERT	67.65	86.03	97.76	96.52	49.02	85.59	91.60	15.78	47.09	33.97	37.54
MulT	65.48	84.72	97.93	96.83	49.72	88.12	92.23	26.12	48.91	34.68	33.95
MISA	63.91	86.63	97.78	98.03	53.44	87.14	92.05	22.15	46.44	36.15	38.74

Results (macro F1-score) of each fine-grained classes in "Achieve goals"

Methods	Inform	Advise	Arrange	Introduce	Comfort	Leave	Prevent	Greet	Ask for help
Human	79.69	87.14	81.40	84.09	95.95	97.06	86.43	94.15	88.54
Text-classifier	67.74	67.68	64.67	68.64	77.05	73.37	82.47	84.90	66.20
MAG-BERT	71.00	69.30	63.82	67.42	76.43	75.77	85.07	91.06	64.44
MulT	70.85	69.43	65.44	71.19	76.44	75.58	81.68	86.65	69.12
MISA	70.18	69.56	67.32	67.22	78.78	77.23	83.30	82.71	67.57