RMinerEvaluationTools

December 18, 2019 · View on GitHub

Tools used for the evaluation of RefactoringMiner 2.0

Oracle

We used the >7000 refactorings in this oracle as the ground truth for evaluating the precision and recall of the 6 tools.

We made the following updates to this tool:

Updated the description format of the reported refactorings to match that of the oracle
Updated the Git checkout process: ** if git checkout for a commit fails, try 'git checkout master -f' and then try to checkout the commit again ** If this does not work, we use the Github API to download the zip of the project at the particular commit
Added Extract And Move method feature by : ** If the 'from' and the 'to' methods belong to different classes, we classify it as Extract And Move Method Refactoring ** Ensuring that no rename, move and move & rename class refactoring led to this conclusion

To evaluate this tool over the oracle:

Download the latest JSON from the oracle and update this file.
Fill in your GitHub credentials here
Provide a path to the folder where repositories could be cloned or found, here
Execute TestAllRefactoringsRD.testAllRefactorings to run the evaluation
Output of the evaluation will be found in the folder named 'Output' in the working directory.
Execute the main method in AnalyseResults to calculate precision and recall per refactoring.

We made the following updates to this tool:

Updated the description format of the reported refactorings to match that of the oracle
Updated the Git checkout process: ** if git checkout for a commit fails, try 'git checkout master -f' and then try to checkout the commit again ** If this does not work, we use the Github API to download the zip of the project at the particular commit
Added Extract And Move method feature by : ** If the 'from' and the 'to' methods belong to different classes, we classify it as Extract And Move Method Refactoring ** Ensuring that no rename, move and move & rename class refactoring led to this conclusion

To evaluate this tool over the oracle:

Download the latest JSON from the oracle and update this file.
Fill in your GitHub credentials here
Provide a path to the folder where repositories could be cloned or found, here
Execute TestAllRefactoringsRD.testAllRefactorings to run the evaluation
Output of the evaluation will be found in the folder named 'Output' in the working directory.
Execute the main method in AnalyseResults to calculate precision and recall per refactoring.

We made the following updates to this tool:

Updated the description format of the reported refactorings to match that of the oracle
Updated the tool to not report Move and/or Rename method refactorings when the method itself is abstract

To evaluate this tool over the oracle:

Download the latest JSON from the oracle and update this file.
Since this version of RefDiff does not require the entire repository, it fetches the changed files from this folder
Execute TestAllRefactoringsRD.testAllRefactorings to run the evaluation
Output of the evaluation will be found in the folder named 'Output' in the working directory.
Execute the main method in AnalyseResults to calculate precision and recall per refactoring.

We made the following updates to this tool:

Updated the description format of the reported refactorings to match that of the oracle
Updated the tool to not checkout each commit, instead query the changes from using the Github API

To evaluate this tool over the oracle:

Download the latest JSON from the oracle and update this file.
Execute TestAllRefactorings.testAllRefactorings to run the evaluation
Output of the evaluation will be found in the folder named 'Output' in the working directory.
Execute the main method in AnalyseResults to calculate precision and recall per refactoring.

To evaluate RefactoringMiner 2.0:

GumTree is used for the evaluation of Rename and Change Type refactoring. To evaluate GumTree 2.1.2:

Download the latest JSON from the oracle and update this file.
Execute TestAllRefactorings.testAllRefactorings to run the evaluation
Output of the evaluation will be found in the folder named 'Output' in the working directory.
Execute the main method in AnalyseResults to calculate precision and recall per refactoring.

RefactoringMiner and RefDiff RefactoringMiner and Gumtree Runtime comparison