README.md

Optimus-2: Multimodal Minecraft Agent with Goal-Observation-
Action Conditioned Policy
CVPR 2025

Zaijing Li^{1 2}, Yuquan Xie¹, Rui Shao^1✉, Gongwei Chen¹,
Dongmei Jiang², Liqiang Nie^1✉

¹Harbin Institute of Technology,Shenzhen    ²Peng Cheng Laboratory, Shenzhen
^✉ Corresponding author  

:new: Updates

• [06/2025] :fire: We released the third generation of Minecraft agent, Optimus-3! Play with Optimus-3 now! We are refactoring Optimus-1 and Optimus-2 to adapt them to the new framework, like Optimus-3.
• [04/2025] :fire: We release the MGOA dataset on Hugging Face.
• [03/2025] :fire: Project page released.
• [02/2025] :fire: Arxiv paper released.

:balloon: Optimus-2 Framework

We propose Optimus-2, a novel Minecraft agent that incorporates a Multimodal Large Language Model (MLLM) for high-level planning, alongside a Goal-Observation-Action Conditioned Policy (GOAP) for low-level control. GOAP contains (1) an Action-guided Behavior Encoder that models causal relationships between observations and actions at each timestep, then dynamically interacts with the historical observation-action sequence, consolidating it into fixed-length behavior tokens, and (2) an MLLM that aligns behavior tokens with open-ended language instructions to predict actions auto-regressively.

:smile_cat: Evaluation results

We report the average rewards (AR) for Atomic Tasks, average success rate (SR) for Long-Horizon Tasks and Open-Ended Instruction Tasks.

Table 1: Main Result of GOAP on Atomic Tasks.

Table 2: Main Result of Optimus-2 on Long-Horizon Tasks.

Table 3: Main Result of Optimus-2 on Open-Ended Instruction Tasks.

:hugs: Citation

If you find this work useful for your research, please kindly cite our paper:

@InProceedings{Li_2025_CVPR,
    author    = {Li, Zaijing and Xie, Yuquan and Shao, Rui and Chen, Gongwei and Jiang, Dongmei and Nie, Liqiang},
    title     = {Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {9039-9049}
}