Results for "Guiding Language Models of Code with Global Context using Monitors"

Summary of Results (Table 1 in the paper)

configuration	('Compilation Rate (CR)', 'score@1')	('Compilation Rate (CR)', 'score@2')	('Compilation Rate (CR)', 'score@3')	('Compilation Rate (CR)', 'score@4')	('Compilation Rate (CR)', 'score@5')	('Compilation Rate (CR)', 'score@6')	('Next Identifier Match (NIM)', 'score@1')	('Next Identifier Match (NIM)', 'score@2')	('Next Identifier Match (NIM)', 'score@3')	('Next Identifier Match (NIM)', 'score@4')	('Next Identifier Match (NIM)', 'score@5')	('Next Identifier Match (NIM)', 'score@6')	('Identifier Sequence Match (ISM)', 'score@1')	('Identifier Sequence Match (ISM)', 'score@2')	('Identifier Sequence Match (ISM)', 'score@3')	('Identifier Sequence Match (ISM)', 'score@4')	('Identifier Sequence Match (ISM)', 'score@5')	('Identifier Sequence Match (ISM)', 'score@6')	('Prefix Match (PM)', 'score@1')	('Prefix Match (PM)', 'score@2')	('Prefix Match (PM)', 'score@3')	('Prefix Match (PM)', 'score@4')	('Prefix Match (PM)', 'score@5')	('Prefix Match (PM)', 'score@6')
CG-350M	35.72	43.04	46.82	49.27	51.04	52.43	68.89	72.68	74.41	75.50	76.30	76.94	26.10	28.74	29.99	30.79	31.38	31.86	21.81	24.12	25.23	25.93	26.45	26.86
CG-350M-MGD	44.58	53.70	58.41	61.47	63.69	65.37	76.22	79.89	81.53	82.55	83.26	83.80	28.15	30.93	32.27	33.14	33.79	34.31	23.38	25.78	26.94	27.69	28.25	28.69
CG-2B	42.05	48.75	52.07	54.21	55.78	57.01	73.81	77.26	78.85	79.86	80.57	81.11	30.20	32.94	34.31	35.21	35.86	36.38	25.52	27.93	29.12	29.91	30.49	30.95
CG-2B-MGD	52.88	61.23	65.23	67.74	69.52	70.91	80.52	83.75	85.21	86.13	86.80	87.32	32.36	35.31	36.78	37.75	38.46	39.03	27.28	29.84	31.11	31.95	32.57	33.06
CG-6B	44.19	50.59	53.77	55.87	57.42	58.64	74.53	77.83	79.33	80.28	80.98	81.55	31.01	33.77	35.12	36.00	36.66	37.17	26.35	28.79	29.95	30.71	31.26	31.69
CG-6B-MGD	55.18	62.92	66.66	69.10	70.88	72.28	81.09	84.14	85.48	86.31	86.90	87.35	33.17	35.99	37.39	38.30	38.99	39.55	28.12	30.58	31.76	32.53	33.10	33.56
SC	44.70	51.32	54.69	56.93	58.63	59.97	75.74	78.91	80.32	81.21	81.87	82.40	31.84	34.59	35.97	36.89	37.58	38.14	26.75	29.10	30.27	31.05	31.63	32.10
SC-MGD	55.34	63.56	67.49	69.94	71.69	73.03	82.30	85.24	86.53	87.34	87.94	88.42	34.14	37.02	38.46	39.41	40.12	40.69	28.60	31.10	32.35	33.17	33.77	34.25
SC-classExprTypes	48.06	55.40	59.04	61.41	63.17	64.57	78.57	81.61	82.96	83.80	84.42	84.91	33.21	36.06	37.48	38.41	39.11	39.67	27.93	30.41	31.64	32.45	33.06	33.55
SC-classExprTypes-MGD	56.80	65.20	69.23	71.77	73.59	75.01	83.55	86.39	87.62	88.39	88.94	89.37	34.92	37.86	39.31	40.27	40.99	41.56	29.30	31.82	33.06	33.88	34.49	34.98
SC-RLPG	50.59	57.76	61.24	63.48	65.11	66.39	79.65	82.51	83.75	84.50	85.03	85.42	35.57	38.52	40.03	41.03	41.77	42.35	30.34	32.93	34.23	35.08	35.71	36.21
SC-RLPG-MGD	60.50	68.78	72.66	75.08	76.80	78.14	84.71	87.28	88.37	89.05	89.53	89.89	37.48	40.56	42.11	43.13	43.88	44.47	31.81	34.53	35.88	36.77	37.44	37.97
SC-FIM	51.72	59.31	62.94	65.25	66.92	68.23	79.58	82.42	83.68	84.49	85.08	85.56	35.53	38.49	39.95	40.91	41.64	42.22	30.34	32.93	34.19	35.02	35.63	36.12
SC-FIM-MGD	62.16	70.73	74.71	77.17	78.89	80.19	84.68	87.24	88.33	89.01	89.50	89.89	37.55	40.61	42.13	43.13	43.89	44.50	31.94	34.60	35.90	36.76	37.40	37.91
SC-FIM-classExprTypes	53.30	61.43	65.35	67.82	69.60	70.97	81.36	84.13	85.32	86.06	86.58	86.99	36.10	39.04	40.48	41.42	42.12	42.67	30.62	33.19	34.44	35.26	35.87	36.36
SC-FIM-classExprTypes-MGD	61.71	70.61	74.70	77.20	78.97	80.33	85.32	87.81	88.89	89.56	90.04	90.42	37.68	40.61	42.02	42.94	43.63	44.18	31.88	34.51	35.79	36.63	37.25	37.75
TD-3	51.71	56.73	59.11	60.64	61.77	62.66	80.76	83.42	84.56	85.27	85.78	86.18	38.57	41.49	42.87	43.77	44.44	44.97	33.13	35.69	36.90	37.70	38.29	38.77
TD-3-MGD	61.23	67.27	70.11	71.93	73.24	74.26	86.32	88.71	89.74	90.37	90.83	91.19	40.69	43.67	45.12	46.07	46.77	47.33	34.29	36.84	38.07	38.87	39.46	39.94

Effect of MGD on Models across Parameter Scale and Architectures (Ref. section 4.1)

Compilation Rate (CR) score@k	Relative Change in Compilation Rate (CR) (score@6)
Next Identifier Match (NIM) score@k	Relative Change in Next Identifier Match (NIM) (score@6)
:-------------------------:	:-------------------------:
Identifier Sequence Match (ISM) score@k	Relative Change in Identifier Sequence Match (ISM) (score@6)
:-------------------------:	:-------------------------:
Prefix Match (PM) score@k	Relative Change in Prefix Match (PM) (score@6)
:-------------------------:	:-------------------------:

Effect of MGD and Prompt Augmentation Strategies (Ref. section 4.2)

Compilation Rate (CR) score@k	Relative Change in Compilation Rate (CR) (score@6)
Next Identifier Match (NIM) score@k	Relative Change in Next Identifier Match (NIM) (score@6)
:-------------------------:	:-------------------------:
Identifier Sequence Match (ISM) score@k	Relative Change in Identifier Sequence Match (ISM) (score@6)
:-------------------------:	:-------------------------:
Prefix Match (PM) score@k	Relative Change in Prefix Match (PM) (score@6)
:-------------------------:	:-------------------------:

Effect of MGD on Fill-in-the-middle (FIM) Decoding (Ref. section 4.3 and appendix E)

Compilation Rate (CR) score@k	Relative Change in Compilation Rate (CR) (score@6)
Next Identifier Match (NIM) score@k	Relative Change in Next Identifier Match (NIM) (score@6)
:-------------------------:	:-------------------------:
Identifier Sequence Match (ISM) score@k	Relative Change in Identifier Sequence Match (ISM) (score@6)
:-------------------------:	:-------------------------:
Prefix Match (PM) score@k	Relative Change in Prefix Match (PM) (score@6)
:-------------------------:	:-------------------------:

Effect of Identifier Complexity on Next Identifier Match (Ref. section 4.4 and appendix F)

Distribution of methods by most complex identifier in DotPrompts

Next Identifier Match (NIM) score@6 by identifier complexity

NIM score@6	Relative Change in NIM (score@6)
NIM score@6	Relative Change in NIM (score@6)
:-------------------------:	:-------------------------:
NIM score@6	Relative Change in NIM (score@6)
:-------------------------:	:-------------------------: