📄Paper: https://dl.acm.org/doi/10.1145/3817119

🏆Leaderboard: Coming Soon

📫Contact: Kaiyuan Liu

📰News

• 2026/07/01 🎉ProjectEvalPLus is accepted by ACM Transactions on Software Engineering and Methodology (TOSEM).

• 2025/11/01 🚀ProjectEvalPlus Repository opensource.

• 2025/05/30 🚀ProjectEval Repository opensource.

• 2025/05/25 🚀ProjectEval Leaderboard online.

• 2025/05/16 🎉ProjectEval is accepted by ACL 2025 Findings.

🚀Quickstart

Requirements

• Operating System: ProjectEvalPlus works on both the Windows platform and the Linux platform. We didn't test macOS yet, but since it works fine on Linux, there should be no problem.

• ❗Browser : ProjectEvalPlus needs a browser and its driver. We officially support three: Edge, Chrome, and Firefox. Now selenium will automatically install driver.

• ❗Python Virtual Environments (venv): ProjectEvalPlus asks you to use a virtual environment by using python -m venv .venv. If your venv path is NOT .venv, change the VENV_PATH to yours in config.ini.

• Java Maven Environments & NodeJS Environments:

• LLM: Make sure that your Ollama runs if you wish to run any models mentioned in our papers.

Automatic Language Extension

WARNING: THIS PHASE WILL INCUR COSTS.

ProjectEvalPlus used GPT-5-mini and GPT-oss-20B to perform ALE. To reperform it:

python run_renew_language.py -l "<Java or JavaScript>" -m "<Model you want use."

Evaluation

Preparation

ProjectEvalPlus standard evaluation only supports JSON. But you can easily transfer any text files into JSON by using tools\file_transform.py .

Execution

If you trust your LLM that it won't do harm to your device, you can run the execution evaluation process by just using:

 python run_judge.py -r "[\"<your_folder_name_in_experiment>\"]" --question_path "data/project_eval_project_<.json"

If NOT, run a Docker by following the following steps:

# Step 1
cd docker

# Step 2
sh build.sh # Linux
build.bat # Windows

# Step 3
sh compose.sh # Linux
compose.bat # Windows

No matter which way you choose, all the results will be saved in the experiments directory.

Objective Indicators

In the project root directory:

python run_indicators.py -r "[\"<your_folder_name_in_experiment>\"]"

The result will be in the experiments directory.

Reasoning

ProjectEvalPlus is an offline evaluation benchmark and its evalutaion phase is complicated and time-costy. So the reasoning phase is separated from the evaluation phase. The reasoning phase only produces JSON or files.

Running a standard ProjectEvalPlus reasoning phase by:

1. Open the run_reasoning.py

2. Follow the instruction in the file by editing your own parameters.

3. python run_reasoning.py

Generation

WARNING: THIS PHASE WILL INCUR COSTS.

ProjectEvalPlus used GPT-4o to generate the data. To reperform it:

python run_generation.py

❓Common Issues Checklist

This part is for some common issues that have been noticed by authors, check this before you submit an issue:

• The answer's path follows the example in the experiments directory.

• The answer's format follows the example in the experiments directory, and if you transfer the files into JSON by yourself, we strongly recommend that you use the script file_transform.py in the tools directory.

• config.ini is set correctly.

• Docker runs properly.

👋Overview

• ProjectEvalPlus is a multi-level and multi-languages benchmark designed to evaluate LLMs and agents on project-level code generation through realistic user interactions. It aims to bridge the gap of lacking the ability to automatically evaluate code from users’ perspective, and also lacking the explainability of the results of LLM agents’ code generation capabilities.

Structure

• ProjectEvalPlus integrates natural language, structured checklists, and code skeletons as 3 different level inputs to simulate diverse development scenarios and support explainable evaluations. And it contains its standard Test Suite and Canonical Answer.

Inputs

• Level 1 - Natural Language Prompt (NL Prompt): the agent will receive one or several natural language sentences to describe the target of the project.

• Level 2 - Natural Language Checklist (NL Checklist): the agent will receive a standard natural language checklist describing the project through the abilities and functions that the project should have.

• Level 3 - Skeleton: the agent will receive a skeleton of the standard answer which contains doc-strings and comments to describe the project inside.

Test Suites

A mission test suite will contain two parts:

• Testcodes: a mission contains several automated evaluation Python functions similar to HumanEval testcases.

• Parameter Description (PD): PD is used for a special kind of parameter alignment. These parameters are required by the matching testcode to achieve the established test goal(s).

Canonical Answer

For every mission we constructed has a canonical solution, beside the canonical code, we also build every PD’s standard answer matching to the canonical code called canonical parameter values.

Construction

Testcode is aligned with Checklist. Parameter Description is aligned with Testcode and Canonical Parameter Values. Canonical Parameter Values is aligned with Canonical Code and use for testcode to get passed.

Automatic Language Extension

ProjectEvalPlus achieved Automatic Language Extension (ALE) for CNC. The ProjectEvalPlus extends the original evaluation test points by providing detailed descriptions of the specific errors detected at each point. This enables ProjectEvalPlus to deliver precise, optimization-oriented feedback to agents, allowing them to gain reinforcement learning capabilities

Evaluation

The evaluation process begins by selecting a specific level from the input and presenting it to the agent. The agent generates solution code. The solution code is then fed back into the same agent along with the parameter description. The agent is tasked with answering the parameter description based on its own solution to produce parameter values (PV). The code is then converted into an executable file, creating a tangible project. PV is a substitute to testcode, and testcode is integrated into the ProjectEvalPlus evaluation machine to obtain the evaluation results.

Summary

ProjectEvalPlus introduces automated evaluation tools and heterogeneous software verification systems, enabling fine-grained comparison of model outputs across semantically equivalent input formats. This provides deeper insight into a model understanding of end-to-end software development.