Our Minecraft Agent

Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks

We propose a Hybrid Multimodal Memory module that integrates structured knowledge and multimodal experiences into the memory mechanism of the agent. On top of it, we introduce a powerful Minecraft agent, Optimus-1, which achieves a 30% improvement over existing agents on 67 long-horizon tasks. ✨

Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy

We propose agent Optimus-2 which incorporates a Multimodal Large Language Model for high-level planning, alongside a Goal-Observation-Action Conditioned Policy (GOAP) for low-level control. Optimus-2 exhibits superior performance across atomic tasks, long-horizon tasks, and open-ended instruction tasks in Minecraft. ✨

Optimus-3: Towards Generalist Multimodal Minecraft Agents with Scalable Task Experts

We propose generalist agent, Optimus-3, endowed with multidimensional capabilities including Captioning, Embodied QA, Planning, Action, Grounding, and Reflection. Our comprehensive evaluation demonstrates that it consistently surpasses existing agents in the Minecraft environment across all assessed dimensions. ✨

---

Table of Contents

• Awesome Policy

  • Visuomotor Policy
  • Goal-conditioned Policy

• Awesome Agent

  • End-to-end Architecture
  • Hierarchical Architecture
  • Code based Agent

• Benchmark

---

Awesome Policy

Visuomotor Policy

Title	Venue	Year	Code	Demo
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos	NeurIPS	2022	Github	-
GROOT: Learning to Follow Instructions by Watching Gameplay Videos	ICLR	2024	Github	Demo

Goal-conditioned Policy

Title	Venue	Year	Code	Demo
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge	NeurIPS	2022	Github	Demo
Mastering Diverse Domains through World Models	Nature	2025	Github	Demo
STEVE-1: A Generative Model for Text-to-Behavior in Minecraft	NeurIPS	2023	Github	Demo
Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction	CVPR	2023	Github	-
MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control	NeurIPS Workshop	2024	Github	Demo
Pre-Training Goal-Based Models for Sample-Efficient Reinforcement Learning	ICLR	2024	Github	-
Vision-Language Models Provide Promptable Representations for Reinforcement Learning	Arxiv	2024	Github	-
Reinforcement Learning Friendly Vision-Language Model for MineCraft	ECCV	2024	Github	-
ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting	CVPR	2025	Github	Demo
GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agents	ICLR	2025	-	-
Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy	CVPR	2025	Github	Demo
ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignment	Arxiv	2025	Github	Demo
Open-World Skill Discovery from Unsegmented Demonstrations	ICCV	2025	Github	Demo

---

Awesome Agent

End-to-end Architecture

Title	Venue	Year	Code	Demo
JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse	Findings of ACL	2025	Github	Demo
Optimus-3: Towards Generalist Multimodal Minecraft Agents with Scalable Task Experts	Arxiv	2025	Github	Demo

Hierarchical Architecture

Title	Venue	Year	Code	Demo
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents	NeurIPS	2023	Github	Demo
Skill Reinforcement Learning and Planning for Open-World Long-Horizon Tasks	NeurIPS Workshop	2023	Github	Demo
Learning from Visual Observation via Offline Pretrained State-to-Go Transformer	NeurIPS	2023	Github	Demo
LLaMA Rider: Spurring Large Language Models to Explore the Open World	NAACL Findings	2024	-	-
JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models	NeurIPS Workshop	2023	Github	Demo
Steve-Eye: Equiping LLM-based Embodied Agents with Visual Perception in Open Worlds	ICLR	2024	Github	Demo
Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks	NeurIPS	2024	Github	Demo
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception	CVPR	2024	Github	Demo
OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents	NeurIPS	2024	Github	Demo
MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory	ICLR	2025	-	Demo
ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting	CVPR	2025	Github	Demo
Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy	CVPR	2025	Github	Demo
BAR: ABackward Reasoning based Agent for Complex Minecraft Tasks	Findings of ACL	2025	Github	-
Metagent-P: A Neuro-Symbolic Planning Agent with Metacognition for Open Worlds	Findings of ACL	2025	-	-
M2PA: A Multi-Memory Planning Agent for Open Worlds Inspired by Cognitive Theory	Findings of ACL	2025	-	-
ADAM: An Embodied Causal Agent in Open-World Environments	ICLR	2025	Github	-
Experience-based Knowledge Correction for Robust Planning in Minecraft	ICLR	2026	Github	-

Code-based Agent

Title	Venue	Year	Code	Demo
Voyager: An Open-Ended Embodied Agent with Large Language Models	NeurIPS	2023	Github	Demo
Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory	Arxiv	2023	Github	-
Creative Agents: Empowering Agents with Imagination for Creative Tasks	UAI	2025	Github	Demo
Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft	CVPR	2024	-	Demo
Odyssey: Empowering Minecraft Agents with Open-World Skills	IJCAI	2025	Github	-
See and Think: Embodied Agent in Virtual Environment	ECCV	2024	Github	Demo
RL-GPT: Integrating Reinforcement Learning and Code-as-policy	NeurIPS	2024	-	Demo
LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence	ICML	2025	-	Demo
Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verification	Arxiv	2024	-	-
WALL-E: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents	Arxiv	2024	Github	-
CausalMACE: Causality Empowered Multi-Agents in Minecraft Cooperative Tasks	Findings of EMNLP	2025	-	-

Benchmark

Title	Venue	Year	Code	Demo
Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks	NeurIPS	2024	Github	Demo
Optimus-3: Towards Generalist Multimodal Minecraft Agents with Scalable Task Experts	Arxiv	2025	Github	Demo
MineNPC-Task: Task Suite for Memory-Aware Minecraft Agents	Arxiv	2026	-	-
PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments	CoG	2025	Github	-
Investigating Feasibility of Large Language Model Agent Collaboration in Minecraft and Comparison with Human-Human Collaboration	IJCNLP-AACL	2025	-	-
MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents	NeurIPS	2025	Github	Demo
VillagerAgent: A Graph-Based Multi-Agent Framework for Coordinating Complex Task Dependencies in Minecraft	Findings of ACL	2024	Github	-