MLPerf Benchmark Results in the MLCommons CMX format
March 13, 2025 ยท View on GitHub
This repository contains compacted and aggregated results of the MLPerf Inference benchmark, MLPerf Training benchmark and TinyMLPerf benchmark in the compact MLCommons Collective Mind format for the Collective Knowledge Playground being developed by the MLCommons taskforce on automation and reproducibility.
The goal is to make it easier for the community to analyze MLPerf results, add derived metrics such as performance/Watt and constraints, generate graphs, prepare reports and link reproducibility reports as shown in these examples:
- Power efficiency to compare Qualcomm, Nvidia and Sima.ai devices
- Reproducibility report for Nvidia Orin
How to import raw MLPerf results to CMX format
Install MLCommons CMX framework.
MLPerf inference benchmark results
Follow this README from the related CM automations script.
You can see aggregated results here.
TinyMLPerf benchmark results
Follow this README from the related CM automations script.
You can see aggregated results here.
MLPerf training benchmark results
Follow this README from the related CM automations script.
You can see aggregated results here.
How to update this repository with new results
Using your own Python script
You can use this repository to analyze, reuse, update and improve MLPerf results compact by calculating and adding derived metrics (performance/watt) or links to reproducibility reports that will be visible at the MLCommons CK playground.
Install MLCommons CMX framework.
Pull the CMX repositories with automation recipes and MLPerf results:
cmx pull repo mlcommons@ck --dir=cm4mlops/cm4mlops
cmx pull repo mlcommons@cm4mlperf-results
Find CM entries with MLPerf inference v3.1 experiments from CMD:
cmx find experiment --tags=mlperf-inference,v4.1
Find CM entries with MLPerf inference v3.1 experiments from Python:
import cmind
r = cmind.access({'action':'find',
'automation':'experiment,a0a2d123ef064bcb',
'tags':'mlperf-inference,v3.1'})
if r['return']>0: cmind.error(r)
lst = r['list']
for experiment in lst:
print (experiment.path)
Using CMX script
We created a sample CM script in this repository that you can use and extend to add derived metrics:
cmx run script "process mlperf-inference results" --experiment_tags=mlperf-inference,v4.1
Copyright
2021-2025 MLCommons
License
Project coordinators
Grigori Fursin and Arjun Suresh.
Contact us
This project is maintained by the MLCommons taskforce on automation and reproducibility, cTuning foundation and cKnowledge.org.