README-extra.md

February 16, 2024 · View on GitHub

This portable CM script provides a unified API and CLI to benchmark ONNX models using the MLPerf loadgen. It measures performance without accuracy using randomly generated inputs. If you need accuracy too, please check official CM automation for MLPerf inference.

Development status

[20240214] ONNX runtime (CPU & GPU) is connected with LoadGen and tested on Ubuntu, Windows and MacOS. See sources.

Prerequisites

Install CM with automation recipes

Install MLCommons CM and pull CM repository with portable automation scripts to benchmark ML Systems:

pip install cmind
cm pull repo mlcommons@ck

Clean CM cache

If you want a "clean" environment, you may want to clean your CM cache as follows:

cm rm cache -f

Set up CM virtual environment

Click if you want to use Python virtual environment

We suggest you to install a python virtual environment via CM though it's not strictly necessary (CM can automatically detect and reuse your Python installation and environments):

cm run script "install python-venv" --name=loadgen

You can also install a specific version of Python on your system via:

cm run script "install python-venv" --name=loadgen --version=3.10.7

By default, CM will be asking users to select one from all detected and installed Python versions including the above one, any time a script with python dependency is run. To avoid that, you can set up the following environment variable with the name of the current virtual environment:

export CM_SCRIPT_EXTRA_CMD="--adr.python.name=loadgen"

The --adr flag stands for "Add to all Dependencies Recursively" and will find all sub-dependencies on other CM scripts

Install dependencies via CM (optional)

Click if you want to install specific versions of dependencies

You can skip this sub-section if you want CM to automatically detect already installed ONNX runtime on your system. Otherwise, follow the next steps to install the latest or specific version of ONNX runtime.

Download LoadGen sources from MLPerf inference benchmark

cm run script "get mlperf inference src" --version=r3.1

Install MLPerf LoadGen

We can now install loadgen via CM while forcing compiler dependency to GCC:

cm run script "get mlperf loadgen"

ONNX, CPU

cm run script "get generic-python-lib _onnxruntime"

cm run script "get generic-python-lib _onnxruntime" --version=1.13.1

cm run script "get generic-python-lib _onnxruntime" --version_min=1.10.0

Benchmark standard MLPerf model

You can use CM variations prefixed by _ to benchmark an official MLPerf model (_resnet50 or _retinanet):

cm run script "python app loadgen-generic _onnxruntime _retinanet" --samples=5
cmr "python app loadgen-generic _onnxruntime _resnet50"

Normally, you should see the following performance report from the loadgen:

Click to open


2022-12-06 16:51:39,279 INFO MainThread - __main__ main: Model: /home/gfursin/CM/repos/local/cache/9c825a0a06fb48e2/resnet50_v1.onnx
2022-12-06 16:51:39,279 INFO MainThread - __main__ main: Runner: inline, Concurrency: 4
2022-12-06 16:51:39,279 INFO MainThread - __main__ main: Results: results/resnet50_v1.onnx/inline
2022-12-06 16:51:39,279 INFO MainThread - __main__ main: Test Started
2022-12-06 16:51:39,399 INFO MainThread - loadgen.harness load_query_samples: Loaded 100 samples
2022-12-06 16:51:55,723 INFO MainThread - loadgen.harness issue_query: Queries issued 550
2022-12-06 16:51:55,725 INFO MainThread - loadgen.harness flush_queries: Queries flushed
2022-12-06 16:51:55,731 INFO MainThread - loadgen.harness unload_query_samples: Unloaded samples
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : Offline
Mode     : PerformanceOnly
Samples per second: 33.6903
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes

================================================
Additional Stats
================================================
Min latency (ns)                : 16325180169
Max latency (ns)                : 16325180169
Mean latency (ns)               : 16325180169
50.00 percentile latency (ns)   : 16325180169
90.00 percentile latency (ns)   : 16325180169
95.00 percentile latency (ns)   : 16325180169
97.00 percentile latency (ns)   : 16325180169
99.00 percentile latency (ns)   : 16325180169
99.90 percentile latency (ns)   : 16325180169

================================================
Test Parameters Used
================================================
samples_per_query : 550
target_qps : 50
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 10000
max_duration (ms): 0
min_query_count : 1
max_query_count : 0
qsl_rng_seed : 0
sample_index_rng_seed : 0
schedule_rng_seed : 0
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
accuracy_log_sampling_target : 0
print_timestamps : 0
performance_issue_unique : 0
performance_issue_same : 0
performance_issue_same_index : 0
performance_sample_count : 100

No warnings encountered during test.

No errors encountered during test.
2022-12-06 16:51:55,753 INFO MainThread - __main__ main: Observed QPS: 33.6903
2022-12-06 16:51:55,753 INFO MainThread - __main__ main: Result: VALID
2022-12-06 16:51:55,753 INFO MainThread - __main__ main: Test Completed

  - Running postprocess ...
  - running time of script "app,loadgen,generic,loadgen-generic,python": 370.87 sec.

Benchmark custom model

You can also specify any custom onnx model file as follows:

cm run script "python app loadgen-generic _onnxruntime" --modelpath=<CUSTOM_MODEL_FILE_PATH>

Benchmark Hugging Face model

cmr "python app loadgen-generic _onnxruntime _custom _huggingface _model-stub.ctuning/mlperf-inference-bert-onnx-fp32-squad-v1.1" --adr.hf-downloader.model_filename=model.onnx

See more examples to download Hugging Face models via CM here.

Benchmark using ONNX CUDA

cm rm cache -f
cmr "python app loadgen-generic _onnxruntime _cuda _retinanet" --quiet
cmr "python app loadgen-generic _onnxruntime _cuda _custom _huggingface _model-stub.ctuning/mlperf-inference-bert-onnx-fp32-squad-v1.1" --adr.hf-downloader.model_filename=model.onnx

These cases worked on Windows and Linux but may require GPU with > 8GB memory:

cmr "python app loadgen-generic _onnxruntime _cuda _custom _huggingface _model-stub.steerapi/Llama-2-7b-chat-hf-onnx-awq-w8" --adr.hf-downloader.model_filename=onnx/decoder_model_merged_quantized.onnx,onnx/decoder_model_merged_quantized.onnx_data --samples=2
cmr "python app loadgen-generic _onnxruntime _cuda _custom _huggingface _model-stub.alpindale/Llama-2-13b-ONNX" --adr.hf-downloader.model_filename=FP32/LlamaV2_13B_float32.onnx --adr.hf-downloader.full_subfolder=FP32 --samples=2
cmr "python app loadgen-generic _onnxruntime _cuda _custom _huggingface _model-stub.Intel/gpt-j-6B-int8-static" --adr.hf-downloader.model_filename=model.onnx --adr.hf-downloader.full_subfolder=. --samples=2

TBD: some cases that are not yet fully supported (data types, input mismatch, etc):

cmr "python app loadgen-generic _onnxruntime _custom _huggingface _model-stub.runwayml/stable-diffusion-v1-5" --adr.hf-downloader.revision=onnx --adr.hf-downloader.model_filename=unet/model.onnx,unet/weights.pb --samples=2
cmr "python app loadgen-generic _onnxruntime _cuda _custom _huggingface _model-stub.microsoft/Mistral-7B-v0.1-onnx" --adr.hf-downloader.model_filename=Mistral-7B-v0.1.onnx,Mistral-7B-v0.1.onnx.data  --samples=2
cmr "python app loadgen-generic _onnxruntime _cuda _custom _huggingface _model-stub.alpindale/Llama-2-7b-ONNX" --adr.hf-downloader.model_filename=FP16/LlamaV2_7B_float16.onnx --adr.hf-downloader.full_subfolder=FP16 --samples=2

Other variations and flags:

You can obtain help about flags and variations from CMD:

cm run script "python app loadgen-generic" --help

Available variations:

  _cpu
  _cuda
  _custom
  _custom,huggingface
  _huggingface
  _model-stub.#
  _onnxruntime
  _pytorch
  _resnet50
  _retinanet

Available flags mapped to environment variables:

  --concurrency  ->  --env.CM_MLPERF_CONCURRENCY
  --ep  ->  --env.CM_MLPERF_EXECUTION_PROVIDER
  --execmode  ->  --env.CM_MLPERF_EXEC_MODE
  --interop  ->  --env.CM_MLPERF_INTEROP
  --intraop  ->  --env.CM_MLPERF_INTRAOP
  --modelpath  ->  --env.CM_ML_MODEL_FILE_WITH_PATH
  --output_dir  ->  --env.CM_MLPERF_OUTPUT_DIR
  --runner  ->  --env.CM_MLPERF_RUNNER
  --samples  ->  --env.CM_MLPERF_LOADGEN_SAMPLES
  --scenario  ->  --env.CM_MLPERF_LOADGEN_SCENARIO

Running this app via Docker

cm docker script "python app loadgen-generic _onnxruntime _custom _huggingface _model-stub.ctuning/mlperf-inference-bert-onnx-fp32-squad-v1.1" --adr.hf-downloader.model_filename=model.onnx  --samples=2 --output_dir=new_results --docker_cm_repo=ctuning@mlcommons-ck

Tuning CPU performance via CM experiment

cm run experiment --tags=loadgen,python,llama2 -- cmr script "python app loadgen-generic _onnxruntime _cuda _custom _huggingface _model-stub.steerapi/Llama-2-7b-chat-hf-onnx-awq-w8" --adr.hf-downloader.model_filename=onnx/decoder_model_merged_quantized.onnx,onnx/decoder_model_merged_quantized.onnx_data --samples=2 --intraop={{CM_OPT_INTRAOP{[1,2,4]}}} --interop={{CM_OPT_INTEROP{[1,2,4]}}} --quiet
cm run experiment --tags=loadgen,python,llama2 -- cmr "python app loadgen-generic _onnxruntime" --modelpath={PATH TO ONNX MODEL} --samples=2 --intraop={{CM_OPT_INTRAOP{[1,2,4]}}} --interop={{CM_OPT_INTEROP{[1,2,4]}}} --quiet