Configuring the python environment

:new: Updates

• [03/2026] :fire: We release the OptimusM4 Dataset on Huggingface.
• [03/2026] :fire: We release the Optimus-3-v2 (Huggingface) and MineSys2 Benchmark.
• [02/2026] :fire: We release the demo video on YouTobe and new version of Optimus-3.
• [06/2025] :fire: We release the Optimus-3-preview on Huggingface.
• [06/2025] :fire: Project page and code released.
• [06/2025] :fire: Arxiv paper released.

:rocket: Optimus-3

Given the task "Craft a diamond sword based on the current inventory", Optimus-3 employs Captioning to perceive and interpret the inventory information, Grounding to select appropriate tools, Planning to generate sub-goals based on available materials, Action to execute these sub-goals sequentially, Reflection to assess the current task state, and Embodied QA to verify whether the task has been successfully completed.

🎮 Play with Optimus-3

We provide an interactive interface that enables users to interact with Optimus-3 in Minecraft in real time through a GUI. You can interact with Optimus-3 through instructions to perform Planning, Long-horizon Actions, Captioning, Embodied QA, and Grounding. This is a framework with a separation between the server and client. You can deploy the model on the server (we strongly recommend a GPU with at least 32GB of VRAM), and then initiate interaction with the server from your local machine at any time. Download the Optimus-3-preview version on Huggingface.

Instructions

Mode	Example	Description
🧠 Planning	get a xxx	Plan the steps before taking actions.
🖼️ Captioning	describe this view	Describe what you see in the current view.
❓ EQA	how many xxx	Answer questions about the environment.
🎯 Grounding	locate the xxx	Locate objects / regions in the view.
🖱️ Action	(click / interact)	Execute the planned actions.

Controls

• ⏸️ Pause: pause to switch tasks.
• 🔄 Reset: reset the environment (agent position will be randomly initialized).

Server

Server are deployed on machines with a GPU with at least 28GB of VRAM.

# install java 8
sudo apt install openjdk-8-jdk
sudo apt install xvfb

# install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# download the repo
git clone https://github.com/JiuTian-VL/Optimus-3.git
cd Optimus-3

# environment setting
uv sync
source .venv/bin/activate
uv pip install -r requirements.txt

# Minestudio setting
# We have made some modifications to the original MineStudio. Please use the version we provided.
cd MineStudio
uv pip install -e .
cd ..

# install LLaMA-Factory
git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
uv pip install -e ".[torch,metrics]"

# install flash-attention
uv pip install flash-attn --no-build-isolation

# download checkpoints
mkdir checkpoint
download Optimus-3 mllm (https://huggingface.co/MinecraftOptimus/Optimus-3) into folder 'checkpoint'
download Optimus-3 action head (https://huggingface.co/MinecraftOptimus/Optimus-3-ActionHead) into folder 'checkpoint'
download Optimus-3 task router (https://huggingface.co/MinecraftOptimus/Optimus-3-Task-Router) into folder 'checkpoint'
download original sentence-bert (https://huggingface.co/efederici/sentence-bert-base) into folder 'checkpoint'

# change the ckpt path
change the optimus3 (actionhead,mllm,task router) checkpoint path in gui_server.py (line 229)
change the optimus3 task router checkpoint path in ./src/minecraftoptimus/model/agent/optimus3.py (line 64)
change the sentence-bert checkpoint path in ./src/minecraftoptimus/model/optimus3/modeling_task_router.py (line 11)

# Communication IP settings
input the ip of your server in gui_server.py (line 459)

Client

The client is deployed on your local machine.

# download the repo
git clone https://github.com/lizaijing/OptimusGUI.git
cd OptimusGUI

# Configuring the python environment
Some basic python packages, e.g., python>=3.11 pyqt6 requests numpy...

# Communication IP settings
input the ip of your server in main.py (line 11) and server/api.py (line 12)

How to run

# start the server
python gui_server.py

# start the client
python main.py

# note 
If you encounter an error about the 'collection', change collections to collections.abc in the corresponding location.
If you encounter an error about the 'model_type', you can change the model_type (line 22) into "qwen2_5_vl" in /checkpoint/Optimus3/config.json

:smile_cat: Evaluation on MineSys2 Benchmark

Download the Optimus-3-v2 version on Huggingface.

# geenrate response in parallel
## change the MODEL path to Optimus-3-v2, you can dowmload it on [Huggingface](https://huggingface.co/MinecraftOptimus/Optimus-3-v2)
bash scripts/optimus3/eval/benchmark_generate.sh

# Evaluation Results
## For the caption and vqa, we employ MLLM as evaluator.
## change the ChatGPT api key in JUDGE_API_KEY, and JUDGE_LLM you like.
bash scripts/optimus3/eval/benchmark_eval.sh

:wrench: Data Generation Pipeline

Given a task pool, we utilize a knowledge graph to generate task plans, forming the planning dataset. These plans are then used as instructions for STEVE-1, which interacts with the environment to produce the action dataset. During this process, we randomly sample images and employ expert models with environmental feedback to generate the captioning, embodied QA, and grounding datasets.

🧩 Framework

A: Overview of Optimus-3. Given observations and instructions, Optimus-3 couples System-1 fast reaction (Action) and System-2 deliberate reasoning (Embodied QA, Planning, Grounding, Reflection) within the Dual-Router Aligned MoE architecture. B: The details of Dual-Router Aligned MoE architecture. Horizontally, Task Router assigns each input to its corresponding task expert together with a shared knowledge expert. Vertically, Layer Router accelerates latency-sensitive action inference by selectively skipping intermediate layers. Both routing decisions are made once before the forward pass. C: Performance comparison of Optimus-3 against current task-specific SOTA agents, GPT-4o, and Qwen2.5-VL.

🌳 Dual-Granularity Reasoning-Aware Policy Optimization

Visualization examples of the task-specific fine-grained reward functions in DGRPO. For the Planning task, we design a Dependency-Aware Synthesis Reward, which treats the item's crafting dependency path as thinking reward and assigns fine-grained step-wise supervision as answer reward. For vision-related tasks, we introduce a Hallucination-Aware Consistency Reward that penalizes hallucinated items in the reasoning process and the final answer.

:smile_cat: Evaluation results

Table 1: Main Result of Optimus-3 on MineSys2 Benchmark.

Table 2: Main Result of Optimus-3 on Long-Horizon Benchmark.

:hugs: Citation

If you find this work useful for your research, please kindly cite our paper:

@article{li2025optimus,
  title={Optimus-3: Dual-Router Aligned Mixture-of-Experts Agent with Dual-Granularity Reasoning-Aware Policy Optimization},
  author={Li, Zaijing and Xie, Yuquan and Shao, Rui and Chen, Gongwei and Guan, Weili and Jiang, Dongmei and Wang, Yaowei and Nie, Liqiang},
  journal={arXiv preprint arXiv:2506.10357},
  year={2025}
}