Open-AoE

2,000 Hours of Smartphone-Collected Egocentric Manipulation Data
with a Complete Data-to-Model Toolchain

English | 简体中文

Open-AoE is a large-scale, real-world egocentric manipulation dataset collected entirely with consumer smartphones. It provides approximately 2,000 hours of first-person video with synchronized hand motion, camera motion, and bilingual atomic-action annotations. The repository connects those signals to visualization, human-to-robot retargeting, robot replay, and model-specific training recipes.

Get the Release

Resource	Access	What you get
Technical report	Read on arXiv	Dataset design, processing, analysis, toolchain, and experiments
Open-AoE-2000H	Hugging Face · ModelScope	Dataset files and distribution information
Data specification	Field-level documentation	Directory layout, schemas, coordinate systems, and validation notes
Capture app	Search for “具身智能数采助手” in major app stores	Smartphone data collection client

The technical report is available on arXiv.

Choose Your Starting Point

I want to…	Start here	Result
Understand what is in one segment	Dataset specification	Learn the video, calibration, MANO, camera-trajectory, and annotation fields
See the data before writing code	AoE-Visualization	Render an end-to-end review video with hands, trajectories, actions, and a 3D view
Train a VLA policy	LeRobot recipes · GR00T N1.7 · H-RDT · VITRA	Convert Open-AoE signals to model-specific state/action semantics and launch training
Train a world or video-action model	Training recipe index	Use DreamZero, LingBot-VA, Ctrl-World, iVideoGPT, GenieRedux, LAOM, AdaWorld, or DreamDojo integrations
Reconstruct scenes or retarget human motion	AoE-Reconstruct-Retarget	Reconstruct interaction assets and produce robot trajectories, simulation renders, or robotized video
Add a model or robot integration	Contributing guide	Follow the repository structure and dependency rules

Quick Start: Inspect One Segment

1. Clone the repository

git clone https://github.com/ant-research/Open-AoE.git
cd Open-AoE

2. Download the data

Choose either Hugging Face or ModelScope, then locate one extracted segment directory. Its expected layout is documented in the data specification.

3. Render the segment

Some visualization and retargeting workflows require MANO model files. Register and download MANO_RIGHT.pkl and MANO_LEFT.pkl from the MANO website, then install them into the repository's shared asset directory:

bash assets/mano/download_mano.sh \
  ~/Downloads/MANO_RIGHT.pkl \
  ~/Downloads/MANO_LEFT.pkl

cd aoe-visualization
pip install -r requirements.txt
python visualize.py --sample /path/to/open_aoe_segment

The output is output/<segment-name>/AoE_output_vis.mp4, a synchronized review of the ego video, reconstructed hands, wrist trajectories, atomic actions, world-frame motion, and timeline. See the visualization guide for EGL/OpenGL requirements and batch rendering.

From Raw Data to Model Input

Each segment is a synchronized multimodal record rather than a standalone video:

Signal	Main artifact	Typical use
Raw and undistorted RGB	raw_video.mp4, raw_video_undistorted.mp4	Visual observation, video modeling, and overlays
Camera metadata	video_info.json, undistorted_video_info.json	Intrinsics, distortion, device information, resolution, and frame rate
Camera motion	camera_traj.npz and transforms in hands.npz	Metric-scale 6-DoF trajectories and world/camera transforms
Hand reconstruction	hands.npz	Per-frame bilateral MANO pose, shape, root transform, and validity
Atomic actions	ego_action_annotation.json	Temporally aligned actions with hand, verb, object, and bilingual description

The toolchain then maps those synchronized signals into task-specific representations:

Stage	Component	Output
Inspect	AoE-Visualization	One review video per segment for visual and temporal quality checks
Reconstruct / retarget	AoE-Reconstruct-Retarget	Reconstructed assets, robot trajectories, simulation validation, and robotized video
Convert / train	AoE-Training-Ready	Model-specific datasets, actions, patches, launchers, and training recipes

Training Recipe Map

Target	Included recipes	Recommended entry
VLA policies	ACT, Diffusion Policy, π0.5, SmolVLA, GR00T N1.7, H-RDT, VITRA	Training-Ready index
World / video-action models	DreamZero, LingBot-VA, Ctrl-World, iVideoGPT	Training-Ready index
Latent-action / world models	GenieRedux, LAOM, AdaWorld, DreamDojo	Training-Ready index

Each recipe is self-contained and documents its upstream repository and verified commit, data conversion, environment variables, training command, outputs, and any required patch. Upstream projects and checkpoints are not vendored into this repository.

Reconstruction and Retargeting Map

Subproject	Coverage	Main capabilities
Phantom	Unitree G1 + Dex3 / Inspire	Arm IK, dexterous-hand retargeting, MuJoCo visualization, and robot overlay
Retarget Galbot	Galbot / Galaxea bimanual platforms	Palm-to-TCP IK, gripper mapping, egoview synthesis, and LeRobot/Rerun export
AoE Retarget Lab	EgoInfinity/G1, Do-as-I-Do/Sharpa, SPIDER/XHand	External-method adapters, 6-DoF reconstruction routes, and a 12-cell comparison matrix

Third-party repositories, model weights, robot assets, and generated videos are not included. Follow each subproject's setup guide to obtain its external dependencies.

Data Processing and Quality Control

1. On-device capture control checks hand visibility, wearing conditions, lighting, motion quality, and device health before upload.
2. Offline quality control and scene labeling filters invalid or sensitive content, standardizes frame rate, slices videos, and assigns scene/task metadata.
3. Reconstruction and annotation estimates camera trajectories, reconstructs MANO hands, and produces atomic-action segments.
4. Quality inspection and delivery applies completeness, correctness, and temporal-consistency gates followed by human review.

Repository Layout

Open-AoE/
├── Open-AoE-tech-report.pdf  # Current technical report
├── open-aoe-2000h/           # Dataset format and field-level documentation
├── aoe-visualization/        # Synchronized data review and rendering
├── aoe-reconstruct-retarget/ # Reconstruction, retargeting, replay, and overlays
├── aoe-training-ready/       # Model adapters, converters, launchers, and patches
├── assets/mano/              # Shared MANO setup helper; model files are not tracked
├── docs/                     # Overview and pipeline figures
├── CONTRIBUTING.md
├── LEGAL.md
└── LICENSE

Why Open-AoE?

• Smartphone-first collection: consumer devices make real-world egocentric capture accessible and scalable.
• Manipulation-aligned signals: calibrated RGB is synchronized with camera motion, MANO hand reconstruction, validity masks, and atomic actions.
• Complete data-to-model path: the repository covers inspection, retargeting, representation conversion, and model-specific training integration.
• Modular and extensible: components can be used independently, and new robot embodiments or model recipes can be added as self-contained integrations.

Citation

The technical report is available on arXiv.

Contact

• Data processing pipeline applications: kaile.zk@antgroup.com
• Industry collaboration: kaile.zk@antgroup.com

Contributing

Contributions of robot embodiments, reconstruction or retargeting backends, visualization features, data converters, and training recipes are welcome. Read CONTRIBUTING.md before opening a pull request.

License and Third-Party Components

Original source code in this repository is released under the Apache License 2.0. Dataset distribution terms, model weights, robot assets, MANO files, and third-party components may use different licenses. Read LEGAL.md and the relevant subproject README before redistribution or commercial use.

Acknowledgements

Open-AoE is built by the AoE community. We gratefully acknowledge the following contributors from industry, universities, and research institutions.

Dataset and Visualization

• National University of Singapore: Qingze Guan
• University of Chinese Academy of Sciences: Zhengxing Wu
• Ant Digital Technology, Ant Group: Zishuo Li, Wanke Zhan, Yang Sun, Zhiyi Huang, Zitong Shan

Open-Source Code, Toolchain, and Experiments

• Zhejiang University: Jiadong Hong
• The University of Hong Kong: Zhenchao Jin, Yushi Feng
• The Hong Kong University of Science and Technology (Guangzhou): Taowen Wang
• Beijing Academy of Artificial Intelligence: You Liu, Yibo Wang
• University of Chinese Academy of Sciences: Yifan Yang
• The Hong Kong University of Science and Technology: Hao Cheng
• Ant Digital Technology, Ant Group: Bowen Yang, Changtao Miao, Zhaowen Zhou, Man Luo

We also thank all data contributors, maintainers, and upstream open-source and research communities whose work made this release possible.