Python Pipeline 性能测试
May 23, 2022 · View on GitHub
测试环境
测试环境如下表所示：
| GPU | 显存 | CPU | 内存 | |
|---|---|---|---|---|
| Serving端 | 4x Tesla P4-8GB | 7611MiB | Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz 48核 | 216G |
| Client端 | 4x Tesla P4-8GB | 7611MiB | Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz 48核 | 216G |
使用单卡GPU，未开启TensorRT。 模型：ResNet_v2_50
性能指标与结论
通过测试，使用 Python Pipeline 模式通过多进程并发，充分利用 GPU 显卡，具有较好的吞吐性能。
测试数据如下：
| model_name | thread_num | batch_size | CPU_util(%) | GPU_memory(mb) | GPU_util(%) | qps(samples/s) | total count | mean(ms) | median(ms) | 80 percent(ms) | 90 percent(ms) | 99 percent(ms) | total cost(s) | each cost(s) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ResNet_v2_50 | 1 | 1 | 2.2 | 3327 | 17.25 | 17.633658869240787 | 355 | 56.428481238996476 | 38.646728515625 | 39.496826171875 | 39.98369140625 | 1273.1911083984373 | 20.131953477859497 | 20.033540725708008 |
| ResNet_v2_50 | 1 | 4 | 2.7 | 3617 | 28.122 | 53.50748430453522 | 268 | 74.71539215543378 | 74.6181640625 | 75.3138671875 | 75.6051025390625 | 77.85322998046874 | 20.03458046913147 | 20.024930953979492 |
| ResNet_v2_50 | 1 | 8 | 1.7 | 3877 | 25.7869 | 59.60582783086999 | 150 | 133.5897119140625 | 132.7503662109375 | 134.968310546875 | 136.470703125 | 140.79039062499996 | 20.132259607315063 | 20.03933620452881 |
| ResNet_v2_50 | 1 | 16 | 7.0 | 4777 | 27.0175 | 63.2627646819339 | 80 | 252.30162048339844 | 251.8448486328125 | 253.046630859375 | 253.91142578125 | 263.361640625 | 20.233070850372314 | 20.18476152420044 |
| ResNet_v2_50 | 1 | 32 | 7.5 | 6567 | 38.532 | 62.945314687348024 | 40 | 506.8969482421875 | 507.3531494140625 | 510.562353515625 | 511.421240234375 | 536.8068920898437 | 20.335111618041992 | 20.276386737823486 |
| ResNet_v2_50 | 2 | 1 | 4.7 | 6567 | 49.4828 | 50.40600094376044 | 1010 | 39.63352195815285 | 39.5345458984375 | 40.452880859375 | 41.1375 | 42.940522460937494 | 20.037296772003174 | 20.01696753501892 |
| ResNet_v2_50 | 2 | 4 | 2.7 | 6567 | 44.4744 | 83.4255836891382 | 420 | 95.38548002697172 | 95.7069091796875 | 97.599951171875 | 98.098583984375 | 102.39680908203125 | 20.137707471847534 | 20.03199553489685 |
| ResNet_v2_50 | 2 | 8 | 2.2 | 6567 | 42.898 | 91.3727510505176 | 230 | 174.89108568274457 | 175.0452880859375 | 175.82001953125 | 176.7634033203125 | 178.64064453125002 | 20.13729453086853 | 20.1132071018219 |
| ResNet_v2_50 | 2 | 16 | 2.2 | 6567 | 45 | 97.5591285698611 | 124 | 327.16720088835683 | 328.6126708984375 | 329.75185546875 | 330.386962890625 | 336.86397460937496 | 20.336385011672974 | 20.284939169883728 |
| ResNet_v2_50 | 2 | 32 | 3.2 | 6567 | 59.5714 | 100.70765418116333 | 64 | 633.9812698364258 | 637.8568115234375 | 648.103515625 | 650.7439697265625 | 659.2212915039062 | 20.336090803146362 | 20.28787398338318 |
| ResNet_v2_50 | 4 | 1 | 3.1 | 6567 | 64.3333 | 80.27845081929433 | 1617 | 49.56464230756223 | 49.4873046875 | 51.5537109375 | 52.693408203125 | 55.207568359374996 | 20.142391681671143 | 20.038144528865814 |
| ResNet_v2_50 | 4 | 4 | 3.3 | 6567 | 70.4563 | 136.62061939701394 | 688 | 116.51574919944586 | 121.8629150390625 | 129.8181640625 | 133.384423828125 | 142.69500732421875 | 20.143372297286987 | 20.041599333286285 |
| ResNet_v2_50 | 4 | 8 | 3.0 | 6567 | 70.896 | 158.46554975132275 | 399 | 201.30669079926378 | 210.69775390625 | 228.51748046875 | 236.427294921875 | 252.24822753906233 | 20.143179416656494 | 20.081032752990723 |
| ResNet_v2_50 | 4 | 16 | 3.2 | 6567 | 66.3832 | 156.4935247130092 | 197 | 407.6668608224937 | 423.974609375 | 450.368212890625 | 464.45986328125 | 482.93658203125 | 20.141408443450928 | 20.078101694583893 |
| ResNet_v2_50 | 4 | 32 | 3.3 | 6567 | 72.4791 | 162.01742190796557 | 104 | 785.5079204852765 | 813.0341796875 | 887.107958984375 | 909.6556640625 | 935.3334838867188 | 20.541000843048096 | 20.423666059970856 |
| ResNet_v2_50 | 8 | 1 | 3.5 | 6567 | 93.977 | 115.9749228558386 | 2337 | 68.5580409078145 | 65.45849609375 | 76.13930664062501 | 83.542041015625 | 91.45666015624998 | 20.15090799331665 | 20.028797417879105 |
| ResNet_v2_50 | 8 | 4 | 4.2 | 6567 | 90.0952 | 175.58748591910316 | 889 | 180.7330482920592 | 170.5810546875 | 218.99931640625 | 240.06337890625002 | 254.413759765625 | 20.252012729644775 | 20.084695398807526 |
| ResNet_v2_50 | 8 | 8 | 2.6 | 6567 | 93.8693 | 206.76595246418208 | 526 | 306.52158695119414 | 303.043212890625 | 321.0791015625 | 350.5477294921875 | 400.32452392578125 | 20.351513147354126 | 20.15437400341034 |
| ResNet_v2_50 | 8 | 16 | 3.2 | 6567 | 85.7273 | 205.31850043117367 | 265 | 614.1745522553066 | 552.372314453125 | 775.89169921875 | 802.022607421875 | 902.2763183593761 | 20.650842428207397 | 20.345011442899704 |
| ResNet_v2_50 | 8 | 32 | 5.0 | 6567 | 89.8717 | 219.8410273718835 | 146 | 1138.4533474020761 | 1039.640869140625 | 1364.289794921875 | 1474.6744384765625 | 1788.2614379882834 | 21.251720190048218 | 20.777225106954575 |
| ResNet_v2_50 | 12 | 1 | 5.0 | 6567 | 89.4762 | 110.00858327847862 | 2218 | 108.50048552943953 | 103.015625 | 121.09404296875003 | 137.1392333984375 | 151.80401123046872 | 20.162063121795654 | 20.055511037508648 |
| ResNet_v2_50 | 12 | 4 | 4.1 | 6567 | 77.7619 | 153.7824464757549 | 779 | 309.68895575507463 | 285.585205078125 | 378.07421875 | 413.481640625 | 424.70853515625 | 20.262390613555908 | 20.104551911354065 |
| ResNet_v2_50 | 12 | 8 | 3.6 | 6567 | 72.6977 | 165.36021780846013 | 425 | 571.1991590073529 | 510.995849609375 | 731.9383300781251 | 747.6568359375 | 757.304716796875 | 20.56117272377014 | 20.230452219645183 |
| ResNet_v2_50 | 12 | 16 | 1.5 | 6567 | 76.2222 | 189.6414991568285 | 252 | 987.7153136238219 | 926.00390625 | 1080.99130859375 | 1249.4956298828126 | 1434.4802392578124 | 21.26116919517517 | 20.74245794614156 |
| ResNet_v2_50 | 12 | 32 | 2.8 | 6567 | 84.25 | 203.868228281784 | 138 | 1811.640237559443 | 1764.2760009765625 | 1855.28046875 | 2023.56826171875 | 2586.8038134765625 | 21.66105055809021 | 20.834286351998646 |
| ResNet_v2_50 | 16 | 1 | 4.8 | 6567 | 94.3333 | 116.34927733312234 | 2347 | 136.7957122373642 | 135.959716796875 | 144.1568359375 | 146.105517578125 | 175.05707519531248 | 20.172020435333252 | 20.067057371139526 |
| ResNet_v2_50 | 16 | 4 | 15.4 | 6567 | 83.6364 | 160.59012047270738 | 822 | 393.3079394412447 | 396.446533203125 | 426.272216796875 | 429.777734375 | 564.1119360351562 | 20.47448492050171 | 20.206754431128502 |
| ResNet_v2_50 | 16 | 8 | 6.8 | 6567 | 81.0233 | 169.95774070621547 | 437 | 741.5512622684854 | 751.521484375 | 763.199169921875 | 948.8041992187501 | 1001.156142578125 | 20.56981921195984 | 20.254074171185493 |
| ResNet_v2_50 | 16 | 16 | 3.5 | 6567 | 77.8706 | 186.56600081516 | 248 | 1332.1007946383568 | 1365.2745361328125 | 1399.212255859375 | 1432.4037353515625 | 1771.4374853515626 | 21.26861262321472 | 20.64799252152443 |
| ResNet_v2_50 | 16 | 32 | 4.3 | 6567 | 83.6371 | 201.1293408638195 | 140 | 2419.3400198800223 | 2561.09228515625 | 2616.081103515625 | 2642.0835205078124 | 2883.8197412109366 | 22.274224042892456 | 21.169659316539764 |