MigrationBench

• 1. 📖 Overview
  • 1.1 MigrationBench: Dataset and Evaluation Framework
  • 1.2 JavaMigration: Migration with LLMs
• 2. 🤗 MigrationBench Datasets
• 3. Code Migration Evaluation
  • 3.1 Docker Mode (Recommended)
    • 3.1.1 Setup Docker
    • 3.1.2 Single Repository Evaluation
    • 3.1.3 Batch Evaluation
  • 3.2 Local Mode
    • 3.2.1 Install Java and Maven
    • 3.2.2 Install MigrationBench
    • 3.2.3 Single Repository Evaluation
    • 3.2.4 Batch Evaluation
  • 3.3 Predictions File Format
• 4. 📚 Citation

1. 📖 Overview

MigrationBench provides an automated and robust framework for evaluating code migration success.

• Reference paper: MigrationBench: Repository-Level Code Migration Benchmark from Java 8

1.1 MigrationBench: Dataset and Evaluation Framework

The name MigrationBench is used for both the dataset and the evaluation framework for code migration success:

1. 🤗 MigrationBench is a large-scale code migration benchmark dataset at the repository level, across multiple programming languages.
   • Current and initial release includes java 8 repositories with the maven build system, as of May 2025.
2. MigrationBench (current Github package) is the evaluation framework to assess code migration success, from java 8 to 17 or any other long-term support (LTS) versions.

The evaluation is an approximation for functional equivalence by checking the following:

1. The repo is able to build and pass all tests
2. Compiled classes' major versions are consistent with the target java version
   • 52 and 61 for java 8 and 17 respectively
3. Test methods are invariant after code migration
4. Number of test cases is non-decreasing after code migration
5. The repos' dependency libraries match their latest major versions
   • Optional for minimal migration by definition, while
   • Required for maximal migration

1.2 JavaMigration: Migration with LLMs

JavaMigration is a separate Github package to conduct code migration with LLMs as a baseline solution, and it relies on the current package for the final evaluation.

2. 🤗 MigrationBench Datasets

There are three datasets in 🤗 MigrationBench:

• All repositories included in the datasets are available on GitHub, under the MIT or Apache-2.0 license.

Index	Dataset	Size	Notes
1	🤗 AmazonScience/migration-bench-java-full	5,102	Each repo has a test directory or at least one test case
2	🤗 AmazonScience/migration-bench-java-selected	300	A subset of 🤗 migration-bench-java-full
3	🤗 AmazonScience/migration-bench-java-utg	4,814	The unit test generation (utg) dataset, disjoint with 🤗 migration-bench-java-full

3. Code Migration Evaluation

MigrationBench supports two evaluation modes:

1. Docker Mode (Recommended): Runs evaluations in isolated Docker containers. No need to install Java or Maven locally.
2. Local Mode: Runs evaluations directly on your machine. Requires Java 17 and Maven 3.9.6 installed locally.

3.1 Docker Mode (Recommended)

Docker mode provides a consistent evaluation environment without requiring local Java/Maven installation. Each evaluation runs in an isolated container, making it ideal for batch processing and reproducible results.

Benefits:

• ✅ No local Java/Maven installation needed
• ✅ Consistent environment across different machines
• ✅ Parallel execution (multiple containers)
• ✅ Easy setup and onboarding

3.1.1 Setup

1. Install Docker:

Follow the official Docker installation guide:

• macOS: https://docs.docker.com/desktop/install/mac-install/
• Windows: https://docs.docker.com/desktop/install/windows-install/
• Linux: https://docs.docker.com/engine/install/

2. Verify Docker:

docker --version

3. Install MigrationBench:

git clone https://github.com/amazon-science/MigrationBench.git
cd MigrationBench

pip install -r requirements.txt -e .

That's it! The Docker image will be built automatically on first run.

3.1.2 Single Repository Evaluation

To run a single repository evaluation in Docker:

GITHUB_URL=https://github.com/0xShamil/java-xid
GIT_DIFF_FILE=/path/to/java-xid.diff

python run_eval.py --github_url $GITHUB_URL --git_diff_filename $GIT_DIFF_FILE --use_docker

Evaluate with a migrated repository directory:

MIGRATED_DIR=/path/to/migrated/repo
python run_eval.py --github_url $GITHUB_URL --migrated_root_dir $MIGRATED_DIR --use_docker

Force rebuild the Docker image:

python run_eval.py --github_url $GITHUB_URL --git_diff_filename $GIT_DIFF_FILE --use_docker --build_docker_image

3.1.3 Batch Evaluation

To run batch evaluations in Docker:

PREDICTIONS=predictions.json

# Run batch evaluation in Docker
python run_eval.py --predictions_filename $PREDICTIONS --use_docker

With parallel processing (recommended for large batches):

# Run 8 Docker containers in parallel (each evaluates one repo)
python run_eval.py --predictions_filename $PREDICTIONS --use_docker --max_workers 8

Important Notes:

• File paths in the predictions file should be absolute paths on your host system
• The Docker container will automatically mount these paths as read-only volumes
• Each repository is evaluated in its own isolated container
• Docker mode automatically handles environment isolation and cleanup

3.2 Local Mode

Local mode runs evaluations directly on your machine. This requires Java 17 and Maven 3.9.6 installed locally.

3.2.1 Install Java and Maven

Install Java 17:

# Install OpenJDK 17
sudo apt update
sudo apt install -y openjdk-17-jdk

# Verify installation
java --version

Install Maven 3.9.6:

# Download and install Maven
curl -O https://archive.apache.org/dist/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.zip
unzip apache-maven-3.9.6-bin.zip
sudo mv apache-maven-3.9.6 /opt/

# Create symlinks
sudo ln -s /opt/apache-maven-3.9.6 /opt/maven # for MAVEN_HOME
sudo ln -s /opt/apache-maven-3.9.6/bin/mvn /usr/local/bin/mvn # so mvn works without PATH setup

# Clean up
rm apache-maven-3.9.6-bin.zip

# Verify installation
mvn --version

Verify installations:

java -version  # Should show version 17
mvn -version   # Should show version 3.9.6

3.2.2 Install MigrationBench

git clone https://github.com/amazon-science/MigrationBench.git
cd MigrationBench

pip install -r requirements.txt -e .

3.2.3 Single Repository Evaluation

GITHUB_URL=https://github.com/0xShamil/java-xid
GIT_DIFF_FILE=/path/to/java-xid.diff

python run_eval.py --github_url $GITHUB_URL --git_diff_filename $GIT_DIFF_FILE

3.2.4 Batch Evaluation

PREDICTIONS=predictions.json

# Sequential processing
python run_eval.py --predictions_filename $PREDICTIONS

# Parallel processing (8 workers)
python run_eval.py --predictions_filename $PREDICTIONS --max_workers 8

3.3 Predictions File Format

For batch evaluation (both Docker and Local modes), provide a predictions file in JSON format.

For each repository, specify the GitHub URL and one of the following:

1. git_diff_file: Path to a file containing the git diff
2. git_diff: Git diff content as a string (can be empty for no changes)
3. migrated_root_dir: Absolute path to the migrated repository directory

Example predictions.json:

[
  {
    "github_url": "https://github.com/0xShamil/java-xid",
    "git_diff_file": "/absolute/path/to/java-xid.diff"
  },
]

or

[
  {
    "github_url": "https://github.com/0xShamil/java-xid",
    "git_diff": "diff --git a/pom.xml b/pom.xml\n--- a/pom.xml\n+++ b/pom.xml\n..."
  },
]

or

[
  {
    "github_url": "https://github.com/0xShamil/java-xid",
    "migrated_root_dir": "/absolute/path/to/migrated/java-xid"
  },
]

4. 📚 Citation

@misc{liu2025migrationbenchrepositorylevelcodemigration,
      title={MigrationBench: Repository-Level Code Migration Benchmark from Java 8},
      author={Linbo Liu and Xinle Liu and Qiang Zhou and Lin Chen and Yihan Liu and Hoan Nguyen and Behrooz Omidvar-Tehrani and Xi Shen and Jun Huan and Omer Tripp and Anoop Deoras},
      year={2025},
      eprint={2505.09569},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2505.09569},
}