train description-java program synthesis on GPU 2 with PLBART

June 22, 2022 · View on GitHub

Code Translation, Summarization & Synthesis

First edit the code/scripts/common_run_utils.sh file, change the ROOT_PATH to your own directory. Then place the unzipped dataset at ROOT_PATH/g4g/XLCoST_data/.

To run the code-to-code translation experiments:

cd code/scripts
# train snippet level java-python translation on GPU 0 with CodeBERT
bash run_translation.sh 0 java python snippet codebert train
# train program level csharp-cpp translation on GPU 2 with PLBART
bash run_translation.sh 2 csharp cpp snippet plbart train
# evaluate program level csharp-cpp translation on GPU 2 with CodeT5
bash run_translation.sh 2 csharp cpp snippet codet5 train

To run the code-to-text and text-to-code generation experiments:

cd code/scripts
# train java-description program summarization on GPU 0 with CodeBERT
bash run_NL_PL.sh 0 java desc program codebert train
# train description-java program synthesis on GPU 2 with PLBART
bash run_NL_PL.sh 2 desc python program plbart eval
# train python-comment snippet summarization on GPU 2 with CodeT5
bash run_NL_PL.sh 2 java comment snippet codet5 eval
# train comment-python snippet synthesis on GPU 2 with CodeT5
bash run_NL_PL.sh 2 comment python snippet codet5 train

Translation Results

Snippet Level

CodeBLEU	Model	C++	Java	Python	C#	JS	PHP	C
C++	Naive Copy	-	64.56	34.79	63.19	53.16	42.56	84.2
	CodeBERT	-	84.94	74.55	84.99	82.79	68.56	45.46
	PLBART	-	83.85	74.89	84.57	83.19	68.62	83.95
	CodeT5	-	86.35	76.28	85.85	84.31	69.87	90.45
Java	Naive Copy	70.85	-	35	78.43	57.81	42.49	69.74
	CodeBERT	87.27	-	58.39	92.26	84.63	67.26	39.94
	PLBART	87.31	-	58.3	90.78	85.42	67.44	72.47
	CodeT5	88.26	-	74.59	92.56	86.22	69.02	82.78
Python	Naive Copy	39.22	31.89	-	31.79	38.34	36.02	37.79
	CodeBERT	80.46	58.5	-	54.72	57.38	65.14	10.7
	PLBART	80.15	74.15	-	73.5	73.2	66.12	62.15
	CodeT5	81.56	78.61	-	78.89	77.76	67.54	68.67
C#	Naive Copy	69.78	78.71	34.77	-	57.85	42.53	66.73
	CodeBERT	86.96	90.15	56.92	-	84.38	67.18	40.43
	PLBART	84.98	6.27	69.82	-	85.02	67.3	75.74
	CodeT5	88.06	91.69	73.85	-	85.95	68.97	81.09
JS	Naive Copy	60.82	59.25	38.84	64.27	-	41.56	55.84
	CodeBERT	84.38	84.42	52.57	84.74	-	66.66	33.29
	PLBART	84.45	84.9	69.29	85.05	-	67.09	72.65
	CodeT5	85.06	85.48	73.15	85.96	-	68.42	80.49
PHP	Naive Copy	36.33	35.61	24.62	36.67	35.55	-	35.95
	CodeBERT	82.58	81.57	69.29	80.96	79.94	-	28.45
	PLBART	83.87	81.66	71.17	78	82.94	-	57.39
	CodeT5	86.33	85.12	73.22	84.56	83.56	-	79.3
C	Naive Copy	83.93	65.46	38.49	63.05	55.55	41.85	-
	CodeBERT	45.84	39.69	13.55	39.71	29.85	38.88	-
	PLBART	82.53	72.35	49.16	75.78	75.05	60.86	-
	CodeT5	90.26	81.81	63.81	83.05	79.73	66.32	-

Program Level

CodeBLEU	Model	C++	Java	Python	C#	JS	PHP	C
C++	Naive Copy	-	57.36	17.68	58.02	53.16	18.97	75.91
	CodeBERT	-	74.73	24.96	76.35	72.95	50.4	21.84
	PLBART	-	75.26	70.13	78.01	61.85	67.01	72.59
	CodeT5	-	80.03	71.56	81.73	79.48	70.44	85.67
Java	Naive Copy	64.25	-	39.87	72.68	57.81	42.51	62.48
	CodeBERT	79.36	-	8.51	84.43	76.02	51.42	21.22
	PLBART	81.41	-	66.29	83.34	80.14	67.12	63.37
	CodeT5	84.26	-	69.57	87.79	80.67	69.44	78.78
Python	Naive Copy	37.47	29.78	-	27.59	38.42	35.48	35.66
	CodeBERT	68.87	28.22	-	17.8	23.65	49.3	18.32
	PLBART	74.38	67.8	-	66.03	69.3	64.85	29.05
	CodeT5	78.85	73.15	-	73.35	71.8	67.5	56.35
C#	Naive Copy	64	73.63	40.09	-	57.79	42.96	60.87
	CodeBERT	78.52	82.25	10.82	-	75.46	51.76	21.63
	PLBART	80.17	81.37	67.02	-	79.81	67.12	57.6
	CodeT5	83.59	85.7	69.52	-	80.5	69.63	77.35
JS	Naive Copy	53.81	51.77	42.31	54.86	-	42.11	49.04
	CodeBERT	75.43	72.33	9.19	75.47	-	52.08	19.79
	PLBART	80.19	76.96	64.18	78.51	-	67.24	67.7
	CodeT5	82.14	79.91	68.42	81.77	-	68.76	74.57
PHP	Naive Copy	34.62	31.33	25.68	32.81	32.26	-	33.45
	CodeBERT	50.13	46.81	16.92	49.75	48.12	-	22.19
	PLBART	79.4	72.77	61.26	74.16	44.26	-	56.23
	CodeT5	85.55	82.09	72.26	83.79	81.72	-	65.86
C	Naive Copy	78.4	59.41	20.2	59.83	53.54	19.75	-
	CodeBERT	21.7	21.27	21.1	19.5	15.64	31.71	-
	PLBART	78.42	13.45	5.53	45.15	31.47	25.17	-
	CodeT5	88.17	**76.

Code Summarization Results

Snippet Level

BLEU	Model	C++	Java	Python	C#	JS	PHP	C
Code Summarization	CodeBERT	14.4	13.13	3.96	14.07	11.81	11.25	5.84
	PLBART	14.77	13.76	8	14.37	10.93	9.07	7.5
	CodeT5	17.36	16.69	10.76	17.44	14.34	13.42	6.63

Program Level

BLEU	Model	C++	Java	Python	C#	JS	PHP	C
Code Summarization	CodeBERT	7.68	5.47	2.04	7.58	7.67	7.5	6.64
	PLBART	7.65	6.35	4.86	9.23	6.78	6.03	4.14
	CodeT5	9.62	8.82	6.32	7.75	8.23	10.5	12.84

Code Synthesis Results

Snippet Level

CodeBLEU	Model	C++	Java	Python	C#	JS	PHP	C
Code Synthesis	CodeBERT	22.7	25.53	12.26	23.44	23.87	36.47	10.63
	PLBART	34.89	32.23	4.62	29.36	29.63	37.56	22.88
	CodeT5	35.48	33.51	21.1	30.64	29.99	36.37	21.93

Program Level

CodeBLEU	Model	C++	Java	Python	C#	JS	PHP	C
Code Synthesis	CodeBERT	26.51	31.14	24.5	33.37	29.09	39.84	18.08
	PLBART	44.09	41.55	33.77	40.7	38.33	43.01	6.72
	CodeT5	45.18	42.73	35.02	43.6	38.66	45.02	34.88