This portable script converts raw results from the TinyMLPerf™ benchmark to the MLCommons CM format for the Collective Knowledge Playground.

The goal is to make it easier for the community to analyze MLPerf inference results, add derived metrics such as performance/Watt and constraints, and link reproducibility reports as shown in these examples:

• Power efficiency to compare Qualcomm, Nvidia and Sima.ai devices
• Reproducibility report for Nvidia Orin

Aggreaged results are available in this MLCommons repository.

You can see these results at [MLCommons CK playground](You can see aggregated results here.

Usage

We have tested this portable CM script on Ubuntu and Windows.

Install MLCommons CM framework.

Pull the MLCommons CK repository with automation recipes for interoperable MLOps:

cm pull repo mlcommons@ck

Install repositories with raw MLPerf inference benchmark results:

cm run script "get git repo _repo.https://github.com/mlcommons/tiny_results_v0.7" --extra_cache_tags=mlperf-tiny-results,version-0.7
cm run script "get git repo _repo.https://github.com/mlcommons/tiny_results_v1.0" --extra_cache_tags=mlperf-tiny-results,version-1.0
cm run script "get git repo _repo.https://github.com/mlcommons/tiny_results_v1.1" --extra_cache_tags=mlperf-tiny-results,version-1.1

You can also add private results to compare submissions locally before they become public:

cm run script "get git repo _repo.https://github.com/mlcommons/submissions_tiny_v1.1" --extra_cache_tags=mlperf-tiny-results,version-1.1-private

You can use a specific checkout/branch as follows:

cm run script "get git repo _repo.https://github.com/mlcommons/submissions_tiny_v1.1" \
   --extra_cache_tags=mlperf-tiny-results,version-1.1-private,generate_final_report \
   --depth="" \
   --branch=generate_final_report

Convert raw MLPerf results into CM experiment entries:

cm run script "import mlperf tiny to-experiment"

Visualize results on your local machine via CK playground GUI:

cm run script "gui _graph" --exp_tags=mlperf-tiny

You can then select the results you want to visualize and compare, add derived metrics and set constaints as shown in the following example:

These results are also available in the public CK playground.

Contact us

This project is maintained by the MLCommons taskforce on automation and reproducibility. Join our Discord server to ask questions, provide your feedback and participate in further developments.