Engine loading Performance Numbers

These numbers were run on a Mac Pro 2019 (2.8 Ghz Intel Core i7)

Because of the warning above, here are some benchmarking numbers to get an idea of the performance of loading in multiple engines vs loading in only one engine.

Benchmarking Numbers for only having one engine loaded in.

	Engine Name	Model	Throughput	Total p50	Total p90	inference p50	inference p90
1 Engine	MXNet	resnet 50	10.19, 5000 iteration / 490602 ms.	95.520 ms	103.606 ms	90.120 ms	97.954 ms
MultiEngine	MXNet	resnet 50	10.17, 5000 iteration / 491656 ms.	96.084 ms	104.198 ms	90.696 ms	98.692 ms
1 Engine	TensorFlow	resnet 50	7.15, 2500 iteration / 349777 ms.	123.493 ms	142.029 ms	110.592 ms	126.268 ms
MultiEngine	TensorFlow	resnet 50	6.84, 2500 iteration / 365355 ms.	124.743 ms	160.793 ms	110.935 ms	144.142 ms
1 Engine	PyTorch	resnet 50	6.87, 5000 iteration / 728242 ms.	140.246 ms	155.400 ms	133.463 ms	147.721 ms
MultiEngine	PyTorch	resnet 50	6.59, 5000 iteration / 759261 ms.	144.358 ms	168.535 ms	137.336 ms	160.346 ms
1 Engine	MXNet	resnet 18	22.04, 5000 iteration / 226893 ms.	43.487 ms	52.077 ms	38.105 ms	45.660 ms
MultiEngine	MXNet	resnet 18	22.64, 5000 iteration / 220807 ms.	42.750 ms	47.923 ms	37.482 ms	42.109 ms
1 Engine	Tensorflow	MobileNet 1.0	30.45, 5000 iteration / 164225 ms.	31.190 ms	34.780 ms	19.983 ms	22.836 ms
MultiEngine	Tensorflow	MobileNet 1.0	29.17, 5000 iteration / 171395 ms.	31.860 ms	38.047 ms	20.631 ms	24.798 ms
1 Engine	PyTorch	resnet 18	14.90, 5000 iteration / 335509 ms.	64.268 ms	73.352 ms	57.620 ms	65.733 ms
MultiEngine	PyTorch	resnet 18	13.54, 5000 iteration / 369255 ms.	71.959 ms	81.688 ms	64.580 ms	73.439 ms