Contents

October 13, 2025 · View on GitHub

MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents

NeurIPS 2025 Datasets and Benchmarks Track 1Shenzhen Campus of Sun Yat-Sen University,  2Shanghai Jiao Tong University, 
3Mohamed bin Zayed University of Artificial Intelligence,  4Huawei Noah’s Ark Lab
* Equal contribution   ✉ Corresponding author  
Spatial Planning is a crucial part in the field of spatial intelligence, which requires the understanding and planning about object arrangements in space perspective. AI agents with the spatial planning ability can better adapt to various real-world applications, including robotic manipulation, automatic assembly, urban planning etc. Recent works have attempted to construct benchmarks for evaluating the spatial intelligence of Multimodal Large Language Models (MLLMs). Nevertheless, these benchmarks primarily focus on spatial reasoning based on typical Visual Question-Answering (VQA) forms, which suffers from the gap between abstract spatial understanding and concrete task execution. In this work, we take a step further to build a comprehensive benchmark called MineAnyBuild, aiming to evaluate the spatial planning ability of open-world AI agents in the Minecraft game. Specifically, MineAnyBuild requires an agent to generate executable architecture building plans based on the given multi-modal human instructions. It involves 4,000 curated spatial planning tasks and also provides a paradigm for infinitely expandable data collection by utilizing rich player-generated content. MineAnyBuild evaluates spatial planning through four core supporting dimensions: spatial understanding, spatial reasoning, creativity, and spatial commonsense. Based on MineAnyBuild, we perform a comprehensive evaluation for existing MLLM-based agents, revealing the severe limitations but enormous potential in their spatial planning abilities. We believe our MineAnyBuild will open new avenues for the evaluation of spatial intelligence and help promote further development for open-world AI agents capable of spatial planning.

motivation

📰 Updates

  • [09/2025] 🎉🎉🎉 Our paper has been accepted as Poster by NeurIPS 2025 Datasets and Benchmarks Track.
  • [05/2025] Arxiv paper released.

Contents

🔧Installation

Install Minecraft

Ensure you have a valid Microsoft account and Minecraft JAVA Edition.

  1. Visit the Minecraft Official Website and navigate to the "Get Minecraft" section.
  2. Select your platform (Windows, Linux or macOS) and choose JAVA Edition.
  3. Download the launcher and install it, and then log in using your Microsoft account.
  4. Start the launcher, select the release version (1.20.4) compatible with Mineflayer and click Play to load the game.
  5. Load the test environment map we provide (recommanded) or create a new world with settings.
  6. Open the Pause Menu, click the Open to LAN, and following the settings:
    • Take note of the Port Number displayed (in the lower versions of Minecraft) or enter a Port Number that is consistent with the code (in the latest versions of Minecraft).
    • Set the game mode to Creative.
    • [IMPORTANT] Enable Cheats to allow command usage required for MLLM-based agents.

For more detailed instructions, visit the Minecraft Help Center for help.

Install packages

  1. Download and install the appropriate version of Node.js.
  2. Run the following command to install Mineflayer on your terminal:
npm install mineflayer

Create a Conda env and packages

conda create -n mineanybuild python=3.10
conda activate mineanybuild
pip install -r requirements.txt

📥Downloads

Download LinksDescription
Hugging FaceDatasets for MineAnyBuild tasks, including task instructions, architectures and stimuli data.
Google DrivePure maps in Maps_Evaluation.zip for evaluation.
Hugging FaceRaw HTML data fetched from Grabcraft, namely Grabcraft_raw_data.rar.

🚀Running codes

📊Evaluation

You should run the Inference of MLLM-based agents first to generate the responses of MLLM-based agents.

For different tasks in our MineAnyBuild, please conduct evaluation according to the following steps respectively.

Executable Spatial Plan Generation, Creativity and Spatial Understanding tasks

  1. Start the Minecraft game and record the video with Replay Mod (recommended currently). 1) Start the recording in Pause Menu of Minecraft game. 2) Run /mineanybuild/mineflayer.ipynb and specify the starting frame. 3) Stop the recording when the notebook cells finish executing. 4) Save and render the video in Replay Mod Menu. (For the concrete instructions, please refer to the Replay Mod documentation and Section B.3.2 in the Supplementary Material. We provide a document for usage instructions of Replay Mod in docs/replay_mod.md.)

  2. Video Segmentation Split the video into multiple frames and match each frame to its corresponding building structure. We provide an example of using this in frame_clipper() and select_frames() functions of /mineanybuild/utils.py. You can use it to match automatically or conduct manual matching selection (recommended for accuracy) if less data is to be tested.

  3. Run the functions/scripts in /mineanybuild/evaluator.py following the below instructions.

    python /mineanybuild/evaluator.py --task [Spatial_Understanding|Creativity|Executable_Spatial_Plan_Generation]

  4. Run the function process_json_critic_scores() in /mineanybuild/utils.py to process the JSON files generated by the previous step.

  5. Run the functions calculate_scores_creativity(), calculate_scores_spatial_plan(), calculate_scores_spatial_understanding() to calculate the final scores of the three tasks respectively.

Spatial Reasoning task

  1. Run the function calculate_scores_spatial_reasoning() in /mineanybuild/evaluator.py to calculate the Accuracy of the Spatial Reasoning task.

Spatial Commonsense task

  1. Run the functions critic_spatial_commonsense() (or critic_spatial_commonsense_opensource()) in /mineanybuild/evaluator.py following the below instructions.

    python /mineanybuild/evaluator.py --task Spatial_Commonsense

  2. Run the function calculate_scores_spatial_commonsense() in /mineanybuild/evaluator.py to calculate the scores of the Spatial Commonsense task.

🗃️Data curation

Executable Spatial Plan Generation, Creativity and Spatial Understanding tasks

For the released data on HuggingFace, you can refer the codes in the dirpath /data_curation, e.g., data_curation/gen_data_released.py.

For data you want to curate by yourself, please follow the below steps.

  1. Prepare your own architectures/buildings/assets in your Minecraft maps. You can follow the template in the example file in /examples/village_input.json to annotate your data.

    "plains_meeting_point_3": {       // architecture name
    "start_pos": [1, 5, -796],    // the X,Y,Z position of the left-bottom block of the architecture
    "end_pos": [11, 13, -786]     // the X,Y,Z position of the right-up block of the architecture
},
...

    By manually annotating the above data, you can directly obtain curated data through the following automated pipeline.

  2. Run /data_curation/customize_data.ipynb to obtain the curated data.

    Optional: If the JSON format is not completely correct, run the function process_json() in /data_curation/utils.py to process the data and format it in JSON.

  3. Run the main function curate_architecture_datasets() for reference in /data_curation/gen_data_customized.py to curate the datasets and generate instructions for different tasks.

  4. [Optional] Calculate the difficulty factor of new data.

    python /data_curation/calculate_difficulty.py

Spatial Reasoning task

For the released data on HuggingFace, you can refer codes in data_curation/gen_data_released.py.

For data you want to curate by yourself, please follow the below steps.

  1. Prepare your own stimuli structures built in your Minecraft map. You can follow the template in the example file in /examples/mental_rotation_input.json to annotate your data.

  2. Run /data_curation/customize_data.ipynb to obtain the curated data.

    Optional: If the JSON format is not completely correct, run the function process_json() in /data_curation/utils.py to process the data and format it in JSON.

  3. Run the main function curate_spatial_reasoning_datasets() for reference in /data_curation/gen_data_customized.py to curate the datasets and generate VQAs data for this task.

Data standardization

Run the functions/scripts in /data_curation/unify_data.py following the below instructions.

python unify_data.py --task [Spatial_Understanding|Creativity|Executable_Spatial_Plan_Generation|Spatial_Reasoning|Spatial_Commonsense]

🤖Inference of MLLM-based agents

Specify your API keys for each models.

Run the following codes to perform inference for MLLM-based agents.

Proprietary MLLMs

  1. Run /mineanybuild/prompter.py specifying the task of MineAnyBuild. Please specify the input and output file/directories for each task function.
python /mineanybuild/prompter.py --task [Spatial_Understanding|Spatial_Reasoning|Creativity|Executable_Spatial_Plan_Generation|Spatial_Commonsense]
  1. Run the json_parser_blueprint() function in /mineanybuild/utils.py to transform the response of MLLM-based agents into blueprint 3D matrix in JSON format.

Open-source MLLMs

  1. Run the following codes based on the type of open-source MLLMs you want to run.
python /mineanybuild/internvl.py --task [Spatial_Understanding|Spatial_Reasoning|Creativity|Executable_Spatial_Plan_Generation|Spatial_Commonsense]
python /mineanybuild/qwenvl.py --task [Spatial_Understanding|Spatial_Reasoning|Creativity|Executable_Spatial_Plan_Generation|Spatial_Commonsense]
python /mineanybuild/llavaov.py --task [Spatial_Understanding|Spatial_Reasoning|Creativity|Executable_Spatial_Plan_Generation|Spatial_Commonsense]
  1. Run the json_parser_blueprint() function in /mineanybuild/utils.py to transform the response of MLLM-based agents into blueprint 3D matrix in JSON format.
✅Planned Future Updates
  • More detailed annotations
  • Docs of instructions for Replay Mod
  • Optimization of the use of visualization tools (timer in Replay Mod code, viewer)
  • RL environment codes for Mineflayer
  • Fix known bugs
  • Debugging and provide adaptation to normal python codes (w/o Jupyter notebook)
  • MineRL/MineDojo codes

Citation

If you find this work useful, please consider citing:

@inproceedings{wei2025mineanybuild,
title={MineAnyBuild: Benchmarking Spatial Planning for Open-world {AI} Agents},
author={Ziming Wei and Bingqian Lin and Zijian Jiao and Yunshuang Nie and Liang Ma and Yuecheng Liu and Yuzheng Zhuang and Xiaodan Liang},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2025}
}

Acknowledgement

Some of the codes are built upon APT. Part of our datasets are built upon GrabCraft, Minecraft Wiki and Raekon. Thanks them for their great works and resources!