README.md

Optimus-2: Multimodal Minecraft Agent with Goal-Observation-
Action Conditioned Policy
CVPR 2025

Zaijing Li^{1 2}, Yuquan Xie¹, Rui Shao^1✉, Gongwei Chen¹,
Dongmei Jiang², Liqiang Nie^1✉

¹Harbin Institute of Technology,Shenzhen    ²Peng Cheng Laboratory, Shenzhen
^✉ Corresponding author  

:new: Updates

• [04/2025] :fire: We release the MGOA dataset on Hugging Face. Codes and ckpt are coming soon.
• [03/2025] :fire: Project page released.
• [02/2025] :fire: Arxiv paper released.

:balloon: Optimus-2 Framework

We propose Optimus-2, a novel Minecraft agent that incorporates a Multimodal Large Language Model (MLLM) for high-level planning, alongside a Goal-Observation-Action Conditioned Policy (GOAP) for low-level control. GOAP contains (1) an Action-guided Behavior Encoder that models causal relationships between observations and actions at each timestep, then dynamically interacts with the historical observation-action sequence, consolidating it into fixed-length behavior tokens, and (2) an MLLM that aligns behavior tokens with open-ended language instructions to predict actions auto-regressively.

:smile_cat: Evaluation results

We report the average rewards (AR) for Atomic Tasks, average success rate (SR) for Long-Horizon Tasks and Open-Ended Instruction Tasks.

Table 1: Main Result of GOAP on Atomic Tasks.

Table 2: Main Result of Optimus-2 on Long-Horizon Tasks.

Table 3: Main Result of Optimus-2 on Open-Ended Instruction Tasks.

:hugs: Citation

If you find this work useful for your research, please kindly cite our paper:

@inproceedings{li2025optimus2,
title={Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy},
author={Li, Zaijing and Xie, Yuquan and Shao, Rui and Chen, Gongwei and Jiang, Dongmei and Nie, Liqiang},
booktitle={2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2025},
organization={IEEE}
}