Test Runner

An automatic unit testing tool for G-TransEval. It reads config in JSON format and generate unit test files of C++, Java, C#, JavaScript and Python.

To run this testing tool, you need following packages on your computer:

• GCC or MinGW with C++20 support
• OpenJDK (or any other JDK) >= 11
• .NET SDK >= 6.0 and dotnet-script
• Python and NodeJS

Here's an example of running unit test on C++:

cd code

python run.py --gen_template --gen_test --run_test \
--lang cpp \
--unit_test_config ../config/l1.json \
--test_file ../gold_answer/l1/test.cc \
--out_name ../results/result.json

Once the script has finished running, you will find a result.json file under results folder, displaying how many samples and which of them have passed unit tests.

Meanings of parameters:
--gen_template : whether to generate test templates according to the input and expected output of each questions in config files.
--gen_test : whether to insert the target methods in test file into the test templates to generate complete runnable test files.
--run_test : whether to run the test files.
--lang : which language to be tested. Currently, you can choose cpp / java / java_2 / csharp  / csharp_2 / python / javascript here.
--unit_test_config : path to config file.
--test_file : path to test file that includes translated methods generated by model.
--out_name : path to place the output JSON file.

The run_all.py in code folder provides an example of testing all the gold answers.

In addition, since we found that different models may translate in different ways (for example, some translate vector<int> in C++ to List<Integer> in Java, while others translate to int[]), two different unit test files are generated for each Java and C# sample and passing one of them are considered correct. Thus, you need to run 2 times with java and java_2(or csharp and csharp_2) lang parameter seperately and merge their results.