BENCHMARKS.md

October 6, 2023 · View on GitHub

This file contains benchmark AUC performance for predicting each pathlogy on 20% of each dataset.

To reproduce these results use the script scripts/model_calibrate.py. This script will take a long time so it will cache the results so you can work with them after the compute is done. The script also calculates the calibration for a model but those outputs can be ignored for this benchmarking. Below is an example of running different models on the PadChest pc dataset and writing the output as markdown.

python model_calibrate.py pc resnet50-res512-all -mdtable
python model_calibrate.py pc chexpert -mdtable
python model_calibrate.py pc jfhealthcare -mdtable
python model_calibrate.py pc densenet121-res224-all -mdtable

Results updated: 07/23/2023

NIH ChestX-ray14

Model Name	Atelectasis	Cardiomegaly	Consolidation	Edema	Effusion	Emphysema	Fibrosis	Hernia	Infiltration	Mass	Nodule	Pleural_Thickening	Pneumonia	Pneumothorax
XRV-ResNet-resnet50-res512-all	0.78	0.90	0.78	0.89	0.86	0.88	0.76	0.92	0.69	0.82	0.76	0.76	0.71	0.84
XRV-DenseNet121-densenet121-res224-all	0.76	0.88	0.77	0.85	0.85	0.73	0.72	0.91	0.68	0.80	0.69	0.74	0.71	0.75
jfhealthcare-DenseNet121	0.76	0.85	0.78	0.87	0.87	-	-	-	-	-	-	-	-	-
CheXpert-DenseNet121-ensemble	0.80	0.88	0.79	0.88	0.87	-	-	-	-	-	-	-	-	-

Google

Model Name	# Params	Lung Opacity	Fracture	Nodule or mass	Pneumothorax
XRV-ResNet-resnet50-res512-all	23,538,642	0.7	0.88	-	0.92
XRV-DenseNet121-densenet121-res224-all	6,966,034	0.92	0.74	-	0.85

RSNA

Model Name	# Params	Lung Opacity	Pneumonia
XRV-ResNet-resnet50-res512-all	23,538,642	0.85	0.87
XRV-DenseNet121-densenet121-res224-all	6,966,034	0.88	0.86

SIIM

Model Name	# Params	Pneumothorax
XRV-ResNet-resnet50-res512-all	23,538,642	0.91
XRV-DenseNet121-densenet121-res224-all	6,966,034	0.79

PadChest

Model Name	# Params	Atelectasis	Cardiomegaly	Consolidation	Edema	Effusion	Emphysema	Fibrosis	Fracture	Hernia	Infiltration	Mass	Nodule	Pleural_Thickening	Pneumonia	Pneumothorax
XRV-ResNet-resnet50-res512-all	23,538,642	0.80	0.94	0.88	0.97	0.95	0.86	0.96	0.86	0.95	0.85	0.85	0.76	0.85	0.81	0.87
XRV-DenseNet121-densenet121-res224-all	6,966,034	0.77	0.93	0.88	0.97	0.95	0.87	0.94	0.70	0.96	0.85	0.85	0.69	0.79	0.82	0.81
jfhealthcare-DenseNet121	12,525,301	0.78	0.89	0.82	0.94	0.96	-	-	-	-	-	-	-	-	-	-
CheXpert-DenseNet121-ensemble		0.82	0.92	0.88	0.97	0.97	-	-	-	-	-	-	-	-	-	-

VinBrain

Model Name	# Params	Atelectasis	Cardiomegaly	Consolidation	Effusion	Infiltration	Lung Opacity	Pleural_Thickening	Pneumothorax
XRV-ResNet-resnet50-res512-all	23,538,642	0.60	0.85	0.91	0.85	0.82	0.71	0.79	0.69
XRV-DenseNet121-densenet121-res224-all	6,966,034	0.67	0.90	0.93	0.87	0.86	0.85	0.84	0.93
jfhealthcare-DenseNet121	12,525,301	0.79	0.81	0.95	0.92	-	-	-	-
CheXpert-DenseNet121-ensemble		0.74	0.89	0.97	0.93	-	-	-	-

CheXpert

Model Name	# Params	Atelectasis	Cardiomegaly	Consolidation	Edema	Enlarged Cardiomediastinum	Fracture	Lung Lesion	Lung Opacity	Effusion	Pleural Other	Pneumonia	Pneumothorax	Support Devices
XRV-ResNet-resnet50-res512-all	23,538,642	0.63	0.84	0.74	0.79	0.5	0.58	0.50	0.71	0.81	-	0.67	0.61	-
XRV-DenseNet121-densenet121-res224-all	6,966,034	0.91	0.91	0.90	0.92	0.78	0.74	0.84	0.87	0.94	-	0.84	0.85	-
jfhealthcare-DenseNet121	12,525,301	0.91	0.89	0.91	0.90	-	-	-	-	0.95	-	-	-	-
CheXpert-DenseNet121-ensemble		0.93	0.91	0.91	0.92	-	-	-	-	0.96	-	-	-	-

MIMIC-CXR (CheXpert labels)

Model Name	Atelectasis	Cardiomegaly	Consolidation	Edema	Enlarged Cardiomediastinum	Fracture	Lung Lesion	Lung Opacity	Effusion	Pleural Other	Pneumonia	Pneumothorax	Support Devices
XRV-DenseNet121-densenet121-res224-all	0.88	0.88	0.91	0.92	0.84	0.74	0.82	0.86	0.92	-	0.82	0.81	-
XRV-ResNet-resnet50-res512-all	0.77	0.78	0.88	0.87	0.50	0.66	0.5	0.77	0.9	-	0.77	0.72	-
jfhealthcare-DenseNet121	0.90	0.82	0.93	0.93	-	-	-	-	0.94	-	-	-	-
CheXpert-DenseNet121-ensemble	0.88	0.84	0.93	0.93	-	-	-	-	0.94	-	-	-	-