▶️ Click to watch full demo video

UnrealZoo

Production-Ready Multi-Agent Embodied AI Training Platform

Photo-realistic virtual environments with 100+ scenes, 10+ agents real-time collaboration

🇺🇸 English | 🇨🇳 中文

🚀 Quick Start • 🌐 Website • 📄 Paper • 📚 Docs •

📖 Overview

UnrealZoo is a rich collection of photo-realistic 3D virtual worlds built on Unreal Engine, designed to reflect the complexity and variability of open worlds. There are various playable entities for embodied AI, including human characters, robots, vehicles, and animals.

Integrated with UnrealCV, UnrealZoo provides a suite of easy-to-use Python APIs and tools for various potential applications, such as data annotation and collection, environment augmentation, distributed training, and benchmarking agents.

💡 This repository provides the gym interface based on UnrealCV APIs for UE-based environments, which is compatible with OpenAI Gym and supports the high-level agent-environment interactions in UnrealZoo.

🔥 What's New

UnrealZoo v3.0 is released! This is our biggest update yet, bringing complete heterogeneous multi-agent collaboration capabilities, out-of-the-box interaction systems, and a comprehensive upgrade to the UnrealCV+ Plugin.

🚀 v3.0 Core Updates

🔌 UnrealCV+ Plugin Upgrade

💡 Solving User Pain Points: Panoramic export, NavMesh path planning, UAV simulation, complete interaction system — UnrealZoo v3.0 is fully supported out-of-the-box

📄 View Full Changelog | 📚 View Notion Technical Docs

📦 Environment Package Download

📜 Development History

View Full Changelog

⚡ 30-Second Quick Start

# 1. Install
pip install -e .

# 2. Set environment path
export UnrealEnv=/path/to/UnrealEnv

# 3. Run multi-agent tracking demo
python example/multi_agent/baseline/multi_random_baseline.py \
  -e UnrealTrack-Map_ChemicalPlant_1-ContinuousColor-v0

💡 Tip: First run requires downloading UE5 environment package (67GB), we recommend using ModelScope China mirror for faster download

🌟 Core Features

🎬 System Interaction Demo

🚗 Vehicle Interaction

🤸 Actions & Interactions

🤖 Diverse Controllable Agents

Drone	Robot Dog	Multi-Agent Collaboration

🎮 Interaction Demo (Example Code)

📹 Video Data Recording

C++ video recording pipeline for efficient large-scale dataset collection

python example/DataRecording/VideoRecordingPipeline.py

Note: Before recording, open the binary and type vget /unrealcv/status to check the port number, ensure it matches the port parameter in the code

python example/tracking/basic/tracking_auto_basic.py \
  -e UnrealTrack-Greek_Island-ContinuousColor-v0

python example/navigation/keyboard/navigation_keyboard_human.py \
  -e UnrealNavigation-Demo_Roof-MixedColor-v0

python example/multi_agent/HeterogeneousCooperation/Aerial-Ground-Cooperative.py \
  -e UnrealTrack-Map_ChemicalPlant_1-ContinuousColor-v0

python example/navigation/keyboard/navigation_keyboard_drone.py \
  -e UnrealNavigation-Demo_Roof-ContinuousColor-v0

🏗️ Technical Architecture

Architecture Overview

• Unreal Engine Environments (Binary): UE5.6 runtime environment containing scenes and playable entities
• UnrealCV+ Server: Plugin built into UE binary, including rendering, data capture, object/agent control, command parsing modules. We optimized the rendering pipeline and command system
• UnrealCV+ Client: Python-based utility functions for launching binaries, connecting to servers, and interacting with UE environments. Uses IPC sockets and batch commands for performance optimization
• OpenAI Gym Interface: Agent-level environment interaction interface, supports task customization via configuration files, includes Gym Wrappers toolkit for environment augmentation and population control

Data Flow

User Algorithm (Python) ←→ Gym Interface ←→ UnrealCV Client ←→ UnrealCV Server ←→ UE5.6 Environment
                                              (Socket/WebSocket)

🌍 Scene Gallery

UE5 Example Scenes

More Scenes: Scene Gallery

---

🎮 UnrealZoo Custom Task Example

3D Spatial Navigation Task

📊 Performance Metrics

🚀 Quick Start

Step 1: Install Dependencies

git clone https://github.com/UnrealZoo/unrealzoo-gym.git
cd unrealzoo-gym
pip install -e .

Step 2: Download Environment Package

Extract to UnrealEnv directory:

export UnrealEnv=/path/to/UnrealEnv

Step 3: Run Demos

# Multi-agent random policy
python example/multi_agent/baseline/multi_random_baseline.py \
  -e UnrealTrack-Map_ChemicalPlant_1-ContinuousColor-v0

# Keyboard control navigation
python example/navigation/keyboard/navigation_keyboard_human.py \
  -e UnrealNavigation-Demo_Roof-MixedColor-v0

💡 Tip: If mouse disappears, press ` (above Tab) to release mouse

📦 Applications

📖 Documentation

🗓️ TODO List

• ✅ Release an all-in-one package of the collected environments
• ✅ Add gym interface for heterogeneous multi-agent co-operation
• ✅ Expand the list of supported interactive actions
• Add more detailed examples for reinforcement learning agents
• Add more detailed examples for large vision-language models

🤝 Contributing & Support

• 🌐 Official Website: http://unrealzoo.site/
• 📧 Contact Email: [[zfw1226@gmail.com, wukui@buaa.edu.cn]]
• 💬 Discussions: GitHub Discussions

If you find this helpful, please give us a ⭐ Star!

📄 Citation

If UnrealZoo helps your research, please cite our ICCV 2025 paper:

@inproceedings{zhong2025unrealzoo,
  title={UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI},
  author={Zhong, Fangwei and Wu, Kui and Wang, Churan and Chen, Hao and Ci, Hai and Li, Zhoujun and Wang, Yizhou},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2025}
}

📜 License & Acknowledgments

This project is open-sourced under Apache 2.0 license.

Acknowledgments:

• UnrealCV — UE-Python communication bridge
• OpenAI Gym — RL environment interface standard
• Unreal Engine — Rendering engine
• Smart Locomotion — Character animation
• Animal Pack — Animal models
• Drivable Car — Vehicle system