[ Back to index ]

Artifact evaluation

This document provides the guidelines to evaluate artifacts at ACM and IEEE conferences.

Overview

Shortly after the artifact submission deadline, the AE committee members will bid on artifacts they would like to evaluate based on their competencies and the information provided in the artifact abstract such as software and hardware dependencies while avoiding possible conflicts of interest.

Within a few days, the AE chairs will make the final selection of evaluators to ensure at least two evaluators per artifact.

Evaluators will then have approximately one month to review artifacts via HotCRP, discuss all encountered issues with the authors and help them fix those issues. Remember that our philosophy of artifact evaluation is not to fail problematic artifacts but to help the authors improve their public artifacts, pass evaluation and improve their Artifact Appendix.

In the end, the AE chairs will decide on a set of the standard ACM reproducibility badges (see below) to award to a given artifact based on all reviews and the authors' responses. Such badges will be printed on the first page of the paper and will be available as metadata in the ACM Digital Library

Authors and reviewers are encouraged to check the AE FAQ and contact chairs and the community via our dedicated AE google group in case of questions or suggestions.

ACM reproducibility badges

Reviewers must read a paper and then thoroughly go through the Artifact Appendix to evaluate shared artifacts. They should then describe their experience at each stage (success or failure, encountered problems and how they were possibly solved, and questions or suggestions to the authors), and give a score on a scale of -1 .. +1:

• +1 if exceeded expectations
• 0 if met expectations (or inapplicable)
• -1 if fell below expectations

Artifacts available

• Are all artifacts related to this paper publicly available?

Note that it is not obligatory to make artifacts publicly available!

The author-created artifacts relevant to this paper will receive an ACM "artifact available" badge only if they have been placed on a publicly accessible archival repository such as Zenodo, FigShare, and Dryad.

A DOI will then be assigned to their artifacts and must be provided in the Artifact Appendix!

Notes:

• ACM does not mandate the use of the above repositories. However, publisher repositories, institutional repositories, or open commercial repositories are acceptable only if they have a declared plan to enable permanent accessibility! Personal web pages, GitHub, GitLab and BitBucket are not acceptable for this purpose.
• Artifacts do not need to have been formally evaluated in order for an article to receive this badge. In addition, they need not be complete in the sense described above. They simply need to be relevant to the study and add value beyond the text in the article. Such artifacts could be something as simple as the data from which the figures are drawn, or as complex as a complete software system under study.
• The authors can provide the DOI at the very end of the AE process and use GitHub or any other convenient way to access their artifacts during AE.

Artifacts functional

• Are all components relevant to evaluation included in the package?
• Well documented? Enough to understand, install and evaluate artifact?
• Exercisable? Includes scripts and/or software to perform appropriate experiments and generate results?
• Consistent? Artifacts are relevant to the associated paper and contribute in some inherent way to the generation of its main results?

Note that proprietary artifacts need not be included. If they are required to exercise the package then this should be documented, along with instructions on how to obtain them. Proxies for proprietary data should be included so as to demonstrate the analysis.

The artifacts associated with the paper will receive an "Artifacts Evaluated - Functional" badge only if they are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation.

We usually ask the authors to provide a small/sample data set to validate at least some results from the paper to make sure that their artifact is functional.

Results reproduced

• Was it possible to validate the key results from the paper using the provided artifacts?

You should report any unexpected artifact behavior to the authors (depends on the type of artifact such as unexpected output, scalability issues, crashes, performance variation, etc).

The artifacts associated with the paper will receive a "Results Reproduced" badge only if the key results of the paper have been obtained in a subsequent study by a person or team other than the authors, using artifacts provided by the author.

Some variation of empirical and numerical results is tolerated. In fact it is often unavoidable in computer systems research - see "how to report and compare empirical results" in the AE FAQ page, the SIGPLAN Empirical Evaluation Guidelines, and the NeurIPS reproducibility checklist.

Since it may take weeks and even months to rerun some complex experiments such as deep learning model training, we are discussing a staged AE where we will first validate that artifacts are functional before the camera-ready paper deadline, and then use a separate AE with the full validation of all experimental results with open reviewing and without strict deadlines. We successfully validated a similar approach at the MLCommons open reproducibility and optimization challenges and there is a related initiative at the NeurIPS conference.

Distinguished artifact award

When arranged by the event, an artifact can receive a distinguished artifact award if it is functional, well-documented, portable, easily reproducible and reusable by the community.

---

This document was prepared by Grigori Fursin with contributions from Bruce Childers, Michael Heroux, Michela Taufer and others. It is maintained by the cTuning foundation.