This portable CM script run submission checker and generates summary report for all Tiny MLPerf results using these native scripts.

Usage

We have tested this portable CM script on Ubuntu and Windows.

Install MLCommons CM framework.

Pull the MLCommons CK repository with automation recipes for interoperable MLOps:

cm pull repo mlcommons@ck

Install repositories with raw MLPerf inference benchmark results:

cmr "get git repo _repo.https://github.com/mlcommons/tiny_results_v0.7" --extra_cache_tags=mlperf-tiny-results,version-0.7
cmr "get git repo _repo.https://github.com/mlcommons/tiny_results_v1.0" --extra_cache_tags=mlperf-tiny-results,version-1.0

You can also add private results to compare submissions locally before they become public:

cmr "get git repo _repo.https://github.com/mlcommons/submissions_tiny_v1.1" --extra_cache_tags=mlperf-tiny-results,version-1.1-private

You can use a specific checkout/branch as follows:

cm run script "get git repo _repo.https://github.com/mlcommons/submissions_tiny_v1.1" \
   --extra_cache_tags=mlperf-tiny-results,version-1.1-private,generate_final_report \
   --depth="" \
   --branch=generate_final_report

Now run this script:

cmr "generate mlperf-tiny report"

It will create `summary-{TinyMLPerf version}.csv' report in your current directory.

You can also specify a version of a repository here:

cmr "generate mlperf-tiny report" --repo_tags=1.1-private

These results are also available in the public CK playground.

Contact us

This project is maintained by the MLCommons taskforce on automation and reproducibility. Join our Discord server to ask questions, provide your feedback and participate in further developments.