OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling

🎉 NEWS

• [2026.3.21] 🔥 OmniWorld-Game with Metric Scale is now released! Check out our latest model Pi3X (an enhanced version of Pi3), which supports metric scale reconstruction!
• [2026.1.26] 🎉 OmniWorld was accepted by ICLR 2026!
• [2026.1.7] Update and release OmniWorld-Game, RH20T, RH20T-Human, Ego-Exo4D, EgoDex, Epic-Kitchens.
• [2025.11.11] The OmniWorld is now live on 🤖 ModelScope!
• [2025.10.15] 🔥 The OmniWorld-Game Benchmark is now live on Hugging Face!
• [2025.10.8] The OmniWorld-HOI4D and OmniWorld-DROID dataset is now live on Hugging Face!
• [2025.9.28] The OmniWorld-CityWalk dataset is now live on Hugging Face!
• [2025.9.21] 🔥 The OmniWorld-Game dataset now includes 5k splits in total on Hugging Face!
• [2025.9.17] 🎉 Our dataset was ranked #1 Paper of the Day on 🤗 Hugging Face Daily Papers!
• [2025.9.16] 🔥 The first 1.2k splits release of OmniWorld-Game is now live on Hugging Face! More data is coming soon, stay tuned!

📝 Open-Source Plan

Dataset	Status	Availability	Domain	# Seq.	FPS	Resolution	# Frames	Depth	Camera	Text	Opt. flow	Fg. masks
OmniWorld-Game	✅ Released	11k / 96k	Simulator	96K	24	1280 × 720	18,515K	🙂	🙂	🙂	🙂	🙂
AgiBot	🔜 Planned	-	Robot	20K	30	640 × 480	39,247K	🙂	✅	✅	❌	🙂
DROID	✅ Released	Full	Robot	35K	60	1280 × 720	26,643K	🙂	✅	🙂	🙂	🙂
RH20T	✅ Released	Full	Robot	109K	10	640 × 360	53,453K	❌	✅	🙂	🙂	🙂
RH20T-Human	✅ Released	Full	Human	73K	10	640 × 360	8,875K	❌	✅	🙂	❌	❌
HOI4D	✅ Released	Full	Human	2K	15	1920 × 1080	891K	🙂	🙂	🙂	🙂	✅
Epic-Kitchens	✅ Released	Full	Human	15K	30	1280 × 720	3,635K	❌	🙂	🙂	❌	🙂
Ego-Exo4D	✅ Released	Full	Human	4K	30	1024 × 1024	9,190K	❌	✅	🙂	🙂	❌
HoloAssist	🔜 Planned	-	Human	1K	30	896 × 504	13,037K	❌	🙂	🙂	🙂	❌
Assembly101	🔜 Planned	-	Human	4K	60	1920 × 1080	110,831K	❌	✅	🙂	🙂	🙂
EgoDex	✅ Released	Full	Human	242K	30	1920 × 1080	76,631K	❌	✅	🙂	❌	❌
CityWalk	✅ Released	Full	Internet	7K	30	1280 × 720	13,096K	❌	🙂	✅	❌	❌
Game-Benchmark	✅ Released	Full	Simulator	-	24	1280 × 720	-	🙂	🙂	🙂	🙂	🙂

---

We will refresh this table whenever a milestone is reached. Your feedback and pull-requests are welcome!

✨ Overview

OmniWorld is a large-scale, multi-domain, and multi-modal dataset specifically designed for 🌍4D world modeling, e.g. 4D geometric reconstruction, future prediction & camera-controlled video generation.

🔑 Key Features

• 📊 Massive Scale: 4000+ hours, 600K+ sequences, 300M+ frames
• 🤖 Diverse Domains: sourced from simulator, robot, human & the Internet
• 🎨 Rich Multi-Modality: depth maps, camera poses, text captions, optical flow & foreground mask

🎮 Introducing OmniWorld-Game

OmniWorld-Game is a newly collected high-quality synthetic subset of the main OmniWorld dataset. It features:

• 📊 Scale: 214 hours, 96K video clips, 18M+ frames
• 🧩 Resolution & Diversity: 720P RGB image captured from a wide range of dynamic game environments
• 🎨 Comprehensive Annotations: cover all annotation types of the OmniWorld dataset

🏆 OmniWorld-Game Benchmark

OmniWorld-Game Benchmark offers 4D world modeling evaluation for 3D Geometric Prediction & Camera Control Video Generation. Key Findings:

• 🚫 Current state-of-the-art approaches still show great limitations in modeling complex 4D environments, based on both quantitative metrics and qualitative results.
• 📈 Fine-tuning existing SOTA methods on OmniWorld leads to significant performance gains across 4D reconstruction and video generation tasks, highlighting the value of our dataset.

💡 Dataset Download

You can download the entire OmniWorld dataset using the following command:

# 1. Install (if you haven't yet)
pip install --upgrade "huggingface_hub[cli]"

# 2. Full download
hf download InternRobotics/OmniWorld \
           --repo-type dataset \
           --local-dir /path/to/DATA_PATH

For downloading specific files (e.g., instead of the full OmniWorld-Game dataset), please refer to the download_specific.py.

For detailed usage, please refer to 🤗 OmniWorld Hugging Face

🚀 Visualize as Point Cloud

This script allows you to convert a scene from OmniWorld-Game dataset into a 3D point cloud for inspection.

1. Prerequisites

Please follow the instructions in the "Dataset Download" section to acquire the OmniWorld-Game dataset.

2. Data Structure

Ensure your data is structured correctly. Each scene directory should contain the following subdirectories and files:

<your-data-path>/b04f88d1f85a/
├─ color/              # RGB frames (.png)
├─ depth/              # 16-bit depth maps
├─ flow/               # flow_u_16.png / flow_v_16.png / flow_vis.png
├─ camera/             # split_*.json (intrinsics + extrinsics)
├─ subject_masks/      # foreground masks (per split)
├─ gdino_mask/         # dynamic-object masks (per frame)
├─ text/               # structured captions (81-frame segments)
├─ droidclib/          # coarse camera poses (if you need them)
├─ fps.txt             # source video framerate
└─ split_info.json     # how frames are grouped into splits

3. Usage

Run the visualize_pcd.py script, providing the path to the scene and the desired split index.

• Metric scale point cloud (real-world units): pass --metadata_csv metadata/omniworld_game_metadata.csv so depth and poses are scaled with the per-scene metric factor from the metadata. The output is split{N}_points_metric.ply.
• Relative scale point cloud (no metric): omit --metadata_csv. The output is split{N}_points.ply.

Examples:

# Metric scale (requires metadata CSV)
python scripts/visualize_pcd.py <your-data-path>/b04f88d1f85a --split_idx 0 --metadata_csv metadata/omniworld_game_metadata.csv

# Relative scale (no metadata)
python scripts/visualize_pcd.py <your-data-path>/b04f88d1f85a --split_idx 0

The point cloud is written under the scene directory (e.g. split0_points_metric.ply or split0_points.ply). You can view it with a 3D viewer like MeshLab.

🏁 Awesome Works using OmniWorld Dataset

Depth Anything 3: Recovering the Visual Space from Any Views

π³: Permutation-Equivariant Visual Geometry Learning

Aether: Geometric-Aware Unified World Modeling

WinT3R: Window-Based Streaming Reconstruction With Camera Token Pool

DeepVerse: 4D Autoregressive Video Generation as a World Model

OmniVGGT: Omni-Modality Driven Visual Geometry Grounded Transformer

📄 License

The OmniWorld dataset is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0). By accessing or using this dataset, you agree to be bound by the terms and conditions outlined in this license, as well as the specific provisions detailed below.

• Special Note on Third-Party Content: A portion of this dataset is derived from third-party game content. All intellectual property rights pertaining to these original game assets (including, but not limited to, RGB and depth images) remain with their respective original game developers and publishers.

• Permitted Uses: You are hereby granted permission, free of charge, to use, reproduce, and share the OmniWorld dataset and any adaptations thereof, solely for non-commercial research and educational purposes. This includes, but is not limited to: academic publications, algorithm benchmarking, reproduction of scientific results.

Under this license, you are expressly forbidden from:

• Using the dataset, in whole or in part, for any commercial purpose, including but not limited to its incorporation into commercial products, services, or monetized applications.

• Redistributing the original third-party game assets contained within the dataset outside the scope of legitimate research sharing.

• Removing or altering any copyright, license, or attribution notices.

The authors of the OmniWorld dataset provide this dataset "as is" and make no representations or warranties regarding the legality of the underlying data for any specific purpose. Users are solely responsible for ensuring that their use of the dataset complies with all applicable laws and the terms of service or license agreements of the original game publishers (sources of third-party content).

For the full legal text of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, please visit: https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode.

Citation

If you find this dataset useful, please cite our paper.

@article{zhou2025omniworld,
      title={OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling}, 
      author={Yang Zhou and Yifan Wang and Jianjun Zhou and Wenzheng Chang and Haoyu Guo and Zizun Li and Kaijing Ma and Xinyue Li and Yating Wang and Haoyi Zhu and Mingyu Liu and Dingning Liu and Jiange Yang and Zhoujie Fu and Junyi Chen and Chunhua Shen and Jiangmiao Pang and Kaipeng Zhang and Tong He},
      journal={arXiv preprint arXiv:2509.12201},
      year={2025}
}