Awesome Embodied Data Pyramid

High-Resolution PDF

This repository accompanies a survey on the data pyramid for robotics and embodied AI. It curates the datasets, data-collection paradigms, simulators, and data-hungry model families reviewed in the survey — spanning real-robot data, UMI (in-the-wild) data, egocentric / ego-exo human data, simulation data, and broad general (web-scale) data — together with the VLA / world-action / embodied-VLM models that consume them. Each entry links to its paper, project page, and code where available.

News 🔥🔥🔥

This project is actively maintained and continuously updated to make embodied data resources easier for the open-source community to discover, access, and download.

• 2026-08-22: Added 20 resources from a July–August 2026 arXiv sweep, spanning contact-rich and industrial real-robot datasets (PRISM, SoftVTBench, RoboInter-Data), EgoData Engineering (7 new human-to-robot conversion works), and visuo-tactile world models used as simulators (ViTacWorld, N0-TWAM, OVTF).
• 2026-08-10: Added N0-Foundation and its OpenNeoData release to the real-robot data collection.
• 2026-08-07: Updated the data accumulation figure; introduced the EgoData Engineering collection with Ego2Robot and EgoEngine; and added the CS2-10k and RekaDaily-10k datasets from Reka AI.
• 2026-07-27: Open-sourced the project website and this GitHub project. You are welcome to use the curated links throughout this repository to access and download the collected resources. Issues and pull requests are encouraged to help keep these resources accessible and useful.

We sincerely thank all open-source dataset creators and contributors for their valuable work and continued support of the community. You can also use task-based search on the project website to find and download datasets.

If you have suggestions for new resources, improvements to methodologies, or corrections for broken links, please don't hesitate to open an issue or submit a pull request. Contributions of all kinds are welcome and greatly appreciated.

Table of Contents

Data Sources

• The Data Pyramid: Data Sources

Real-Robot Data

• Section: Real-Robot Data
• Table: Recent real-robot datasets
• Table: Real-robot datasets (2024 and earlier)

UMI Data

• Section: UMI Data
• Table: UMI-style robot-free and cross-embodiment datasets

Egocentric & Ego-Exo Data

• Section: Egocentric & Ego-Exo Data
• Table: EgoData Engineering
• Table: Recent egocentric and ego-exo datasets
• Table: Egocentric and ego-exo datasets (2024 and earlier)

Simulation Data

• Section: Simulation Data
• Table: 3D assets and scene datasets
• Table: Simulation and rendering backends
• Table: Simulation benchmarks
• Table: Simulation/synthetic datasets and data-generation systems
• Table: World models as simulators

General Data

• Section: General Data
• Table: Visual data
• Table: Segmentation and localization data
• Table: Video and temporal data
• Table: Planning and task decomposition data
• Table: Grasp data

Embodied Foundation Models

• Section: Embodied Foundation Models
• Table: Representative VLA and WAM methods

Repository

• Contributing
• Acknowledgement

The Data Pyramid: Data Sources

Real-Robot Data

Table: Recent real-robot datasets

Year	Dataset	Paper	Scale	Venue	Project	Repo@GitHub
2026	PRISM	PRISM: Precision and contact-rich Real-world Industrial Skill dataset with Multimodal sensing	5K+ trajectories / 45 hours	arXiv
2026	SoftVTBench	SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation	4,000 demonstrations	arXiv
2026	RoboInter-Data	RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation	230K+ episodes / 571 scenes	arXiv
2026	PanoVLA-Dataset	Learning Panorama-Aware VLA for Mobile Manipulation with Whole-Body Teleoperation	5.5 hours	arXiv
2026	Handwritten-Alphabet-LfD	Robot Learning from Human Demonstrations: Handwritten Alphabet Trajectories and Human-Likeness Evaluation	3,142 demonstrations	arXiv
2026	OpenNeoData	N0-Foundation: Towards the Age of Tactile Intelligence	200K+ trajectories / 5,000+ hours	Technical Report
2026	ABC-130K	Scalable Behavior Cloning with Open Data, Training, and Evaluation	130.7K trajectories
2026	AgiBot World 2026	AgiBot World 2026: Real-World Embodied Intelligence Dataset	20.7K
2026	DECO-50	DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter	8K trajectories	ICML
2026	Dexora	Dexora: Open-source VLA for High-DoF Bimanual Dexterity	12.2K trajectories	ICRA
2026	HapTile	HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning	1.7K trajectories	arXiv
2026	LET-Base-Dataset	LET-Base-Dataset: LET Basic Operation Dataset	92.6K trajectories
2026	MolmoAct2-BimanualYAM	MolmoAct2: Action Reasoning Models for Real-world Deployment	34.5K trajectories	arXiv
2026	MolmoAct2-SO100/101	MolmoAct2: Action Reasoning Models for Real-world Deployment	38K trajectories	arXiv
2026	OmniViTac	OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation	21.9K trajectories	arXiv
2026	Open-H-Embodiment	Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics	125.8K trajectories	arXiv
2026	Baihu-VTouch	VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation		arXiv
2026	Unitree UnifoLM-WBT	UniFoLM-WBT-Dataset
2025	Aist-Bimanual	AIST Bimanual Manipulation Dataset	10.7K trajectories
2025	PH2D	Humanoid Policy ~ Human Policy	1.6K trajectories	CoRL
2025	ActionNet	ActionNet: A Dataset for Dexterous Bimanual Manipulation	30K trajectories
2025	AgiBot World Beta	AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems	1M trajectories	IROS
2025	Open Galaxea	Galaxea Open-World Dataset and G0 Dual-System VLA Model	20.5K trajectories	arXiv
2025	Humanoid Everyday	Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation	10.3K trajectories	ICRA
2025	MotionTrans	MotionTrans: Human VR Data Enable Motion-Level Learning for Robotic Manipulation Policies	1.5K trajectories	ICRA
2025	RealSource-World	RealSource-World: A Large-Scale Real-World Dual-Arm Manipulation Dataset	26.7K trajectories
2025	REASSEMBLE	REASSEMBLE: A Multimodal Dataset for Contact-rich Robotic Assembly and Disassembly	4.6K trajectories	RSS
2025	RoboCOIN	RoboCOIN: An Open-Sourced Bimanual Robotic Data Collection for Integrated Manipulation	180K trajectories	arXiv
2025	RoboMIND 2.0	RoboMIND 2.0: A Multimodal, Bimanual Mobile Manipulation Dataset for Generalizable Embodied Intelligence	310K trajectories	arXiv

Table: Real-robot datasets (2024 and earlier)

Year	Dataset	Paper	Scale	Venue	Project	Repo@GitHub
2024	FMB	FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning	22.6K trajectories	IJRR
2024	ALOHA Unleashed	ALOHA Unleashed: A Simple Recipe for Robot Dexterity	26.2K trajectories	CoRL
2024	DROID	DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset	76K trajectories	RSS
2024	HumanPlus	HumanPlus: Humanoid Shadowing and Imitation from Humans	240 trajectories	CoRL
2024	Mobile ALOHA	Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation	276 trajectories	CoRL
2024	OmniH2O	OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning		CoRL
2024	Open X-Embodiment	Open X-Embodiment: Robotic Learning Datasets and RT-X Models	2.4M trajectories	ICRA
2024	RoboMIND	RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation	107K trajectories	RSS
2023	ALOHA	Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware	300 trajectories	RSS
2023	RH20T	RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot	110K trajectories	ICRA
2023	RoboSet	RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking	98.5K trajectories	ICRA
2023	BridgeData V2	BridgeData V2: A Dataset for Robot Learning at Scale	60.1K trajectories	CoRL
2023	FurnitureBench	FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation	5.1K trajectories	RSS
2022	BC-Z	BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning	26K trajectories	CoRL
2022	RT-1	RT-1: Robotics Transformer for Real-World Control at Scale	130K trajectories	RSS
2021	MT-Opt	MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale	800K trajectories	CoRL
2021	BridgeData	Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets	7.2K trajectories	RSS
2019	RoboNet	RoboNet: Large-Scale Multi-Robot Learning	162K trajectories	CoRL
2018	DAML	One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning	2.9K trajectories	RSS
2018	MIME	Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation	8.3K trajectories	CoRL
2018	QT-Opt	QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation	580K trajectories	CoRL
2018	RoboTurk	RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation	2.1K trajectories	CoRL
2015	Pinto and Gupta	Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours	50K trajectories	ICRA

UMI Data

Table: UMI-style robot-free and cross-embodiment datasets

Year	Dataset	Paper	Scale	Venue	Project	Repo@GitHub
2026	HuMI	Humanoid Manipulation Interface: Humanoid Whole-Body Manipulation from Robot-Free Demonstrations	827 trajectories	arXiv
2026	RealOmni	RealOmni-Open DataSet	13,000+ hours; 5M+ clips (snapshot: 2026-07-24)
2026	UMI-3D	UMI-3D: Extending Universal Manipulation Interface from Vision-Limited to 3D Spatial Perception	4609 trajectories	arXiv
2026	TAMEn	TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks	724 trajectories	arXiv
2026	Daimon-Infinity	Daimon-Infinity	274669 trajectories
2025	Data Scaling Laws	Data Scaling Laws in Imitation Learning for Robotic Manipulation	24098 trajectories	ICLR
2025	LEGATO	LEGATO: Cross-Embodiment Imitation Using a Grasping Tool	900 trajectories	RA-L
2025	ViTaMIn	ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface	841 trajectories	arXiv
2025	DexWild	DexWild: Dexterous Human Interactions for In-the-Wild Robot Policies	9505 trajectories	RSS
2025	DexUMI	DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation	1792 trajectories	CoRL
2025	FreeTacMan	FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation	>10,000 trajectories; 50 tasks	ICRA
2025	Touch in the Wild	Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper	2737 trajectories	NeurIPS
2025	exUMI	exUMI: Extensible Robot Teaching System with Action-aware Task-agnostic Tactile Representation	1660 trajectories	CoRL
2025	MV-UMI	MV-UMI: A Scalable Multi-View Interface for Cross-Embodiment Learning	1370 trajectories	arXiv
2025	ManipForce	ManipForce: Force-Guided Policy Learning with Frequency-Aware Representation for Contact-Rich Manipulation	597 trajectories	ICRA
2025	FastUMI-100K	FastUMI-100K: Advancing Data-driven Robotic Manipulation with a Large-scale UMI-style Dataset	100K trajectories	arXiv
2025	ViTaMIn-B	ViTaMIn-B: A Reliable and Efficient Visuo-Tactile Bimanual Manipulation Interface	844 demonstrations	arXiv
2024	UMI	Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots	2543 trajectories	RSS
2024	ManiWAV	ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data	1014 trajectories	CoRL
2024	UMI-on-Legs	UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers	514 trajectories	CoRL
2024	FastUMI	FastUMI: A Scalable and Hardware-Independent Universal Manipulation Interface with Dataset	8965 trajectories	CoRL

Egocentric & Ego-Exo Data

Table: EgoData Engineering

Year	Acronym	Paper	Venue	Project	Repo@GitHub
2026	Pegasus	From Passive Video to Editable Experience: Physically Grounded Experience Synthesis for Embodied Intelligence	arXiv
2026	RoboReact	RoboReact: Agentic Skill Distillation from Generated Egocentric Videos for Generalizable Whole-Body Manipulation	arXiv
2026	SiMDex	SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulation	arXiv
2026	HandEdit	HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing	arXiv
2026	VLAff	VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances	arXiv
2026	DreamHand	DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery	arXiv
2026	EgoTac	EgoTac: In-the-wild Tactile Prediction from Egocentric Vision	arXiv
2026	Ego2Robot	Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data	arXiv
2026	EgoEngine	EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations	arXiv
2025	HaWoR	HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos	CVPR

Table: Recent egocentric and ego-exo datasets

Year	Acronym	Paper	Venue	Project	Repo@GitHub
2026	EgoSuite-Open100K	EgoSuite-Open100K: 100,000 Hours of Egocentric Human Data for Physical AI
2026	ACE-Data-0	ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine	arXiv
2026	MOVE	DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents	arXiv
2026	RekaDaily-10k	RekaDaily-10k: Collecting 10,000+ Hours of Egocentric Household Manipulation Data
2026	CS2-10k	CS2-10k: A Large-Scale Egocentric Counter-Strike 2 Dataset
2026	HumanCLAW	HumanCLAW: Can Vision-Language Models Act Through a Body?	arXiv
2026	EgoHTR	EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal	arXiv
2026	EgoSteer	EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos	arXiv
2026	Open-AoE	Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning	arXiv
2026	EgoTactile	EgoTactile: Learning Grasp Pressure for Everyday Objects from Egocentric Video	ICML
2026	EgoEMG	EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation	arXiv
2026	EgoEVHands	EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras	arXiv
2026	EgoSPT	Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulation	arXiv
2026	EgoTraj	EgoTraj: Real-World Egocentric Human Trajectory Dataset for Multimodal Prediction	arXiv
2026	HumanEgo	HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos	arXiv
2026	MobileEgo Anywhere	MobileEgo Anywhere: Open Infrastructure for long horizon egocentric data on commodity hardware	arXiv
2026	EgoTouch	TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video	arXiv
2026	ChildLens	ChildLens: An egocentric video dataset for activity analysis in children	Behavior Research Methods
2026	EgoFun3D	EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates	arXiv
2026	EgoLive	EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks	arXiv
2026	EgoMAGIC	EgoMAGIC: An Egocentric Video Field Medicine Dataset for Training Perception Algorithms	arXiv
2026	EgoVerse	EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World	arXiv
2026	Ego-1K	Ego-1K -- A Large-Scale Multiview Video Dataset for Egocentric Vision	CVPR
2026	EgoXtreme	EgoXtreme: A Dataset for Robust Object Pose Estimation in Egocentric Views under Extreme Conditions	CVPR
2026	FEEL	FEEL (Force-Enhanced Egocentric Learning): A Dataset for Physical Action Understanding	arXiv
2026	Xperience-10M	Xperience-10M: Ropedia Dataset Release
2025	Ego-EXTRA	Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance	arXiv
2025	Egocentric-100K	Egocentric-100K Dataset Release
2025	World In Your Hands	World In Your Hands: A Large-Scale and Open-source Ecosystem for Learning Human-centric Manipulation in the Wild	arXiv
2025	IndEgo	IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants	NeurIPS
2025	OpenEgo	OpenEgo: A Large-Scale Multimodal Egocentric Dataset for Dexterous Manipulation	arXiv
2025	EgoDex	EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video	ICLR
2025	EgoExOR	EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding	NeurIPS
2025	EgoLife	EgoLife: Towards Egocentric Life Assistant	CVPR
2025	TASTE-Rob	TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulation	CVPR
2025	HD-EPIC	HD-EPIC: A Highly-Detailed Egocentric Video Dataset	CVPR
2025	EgoMe	EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real World	arXiv

Table: Egocentric and ego-exo datasets (2024 and earlier)

Year	Acronym	Paper	Venue	Project	Repo@GitHub
2024	EgoVid-5M	EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation	NeurIPS
2024	EgoMimic	EgoMimic: Scaling Imitation Learning via Egocentric Video	ICRA
2024	EgoOops	EgoOops: A Dataset for Mistake Action Detection from Egocentric Videos Referring to Procedural Texts	ICCV Workshops
2024	EgoPressure	EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision	CVPR
2024	EgoExo-Fitness	EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding	ECCV
2024	HOT3D	HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos	CVPR
2024	Nymeria	Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild	ECCV
2024	EgoSurgery	EgoSurgery-Phase: A Dataset of Surgical Phase Recognition from Egocentric Open Surgery Videos	MICCAI
2024	EgoExoLearn	EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World	CVPR
2024	AEA	Aria Everyday Activities Dataset	arXiv
2023	CaptainCook4D	CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities	NeurIPS
2023	Ego-Exo4D	Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives	IJCV
2023	PVSG	Panoptic Video Scene Graph Generation	CVPR
2023	IndustReal	IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting	WACV
2023	EgoObjects	EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understanding	ICCV
2023	ENIGMA-51	ENIGMA-51: Towards a Fine-Grained Understanding of Human-Object Interactions in Industrial Scenarios	arXiv
2023	HoloAssist	HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World	ICCV
2023	ADT	Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception	ICCV
2023	AssemblyHands	AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimation	CVPR
2023	WEAR	WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognition	IMWUT
2022	Touch and Go	Touch and Go: Learning from Human-Collected Vision and Touch	NeurIPS
2022	EgoHOS	Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications	ECCV
2022	ARCTIC	ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation	CVPR
2022	Assembly101	Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities	CVPR
2022	EgoPAT3D	Egocentric Prediction of Action Target in 3D	CVPR
2022	HOI4D	HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction	CVPR
2021	EgoBody	EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices	ECCV
2021	Ego-Deliver	Ego-Deliver: A Large-Scale Dataset for Egocentric Video Analysis	ACM MM
2021	Ego4D	Ego4D: Around the World in 3,000 Hours of Egocentric Video	CVPR
2021	H2O	H2O: Two Hands Manipulating Objects for First Person Interaction Recognition	ICCV
2020	MECCANO	The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-Like Domain	WACV
2020	LEMMA	LEMMA: A Multi-View Dataset for Learning Multi-Agent Multi-Task Activities	ECCV
2020	EPIC-KITCHENS-100	Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100	IJCV
2020	EGO-CH	EGO-CH: Dataset and Fundamental Tasks for Visitors Behavioral Understanding Using Egocentric Vision	PRL
2020	EgoCom	EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset	TPAMI
2019	You2Me	You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions	CVPR
2018	Charades-Ego	Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos	arXiv
2018	EPIC-KITCHENS	Scaling Egocentric Vision: The EPIC-KITCHENS Dataset	ECCV
2018	EgoCart	Egocentric Shopping Cart Localization	ICPR
2018	EgoGesture	EgoGesture: A New Dataset and Benchmark for Egocentric Hand Gesture Recognition	TMM
2018	EGTEA Gaze+	In the Eye of the Beholder: Gaze and Actions in First Person Video	TPAMI
2017	FPHA	First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations	CVPR
2017	EgoDexter	Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor	ICCV
2017	THU-READ	Action Recognition in RGB-D Egocentric Videos	ICIP
2016	EgoConvNet	First Person Action Recognition Using Deep Learned Descriptors	CVPR
2016	KrishnaCam	KrishnaCam: Using a Longitudinal, Single-Person, Egocentric Dataset for Scene Understanding Tasks	WACV
2016	PEV	Recognizing Micro-Actions and Reactions from Paired Egocentric Videos	CVPR
2015	EgoHands	Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions	ICCV
2008	CMU-MMAC	Guide to the Carnegie Mellon University Multimodal Activity (CMU-MMAC) Database	Technical Report

Simulation Data

Curation note. For peer-reviewed work, Year and Publication / Type use the formal publication year and venue. For arXiv-only work, Year is the first public preprint year. Software without a canonical paper is labeled Software. The sections below distinguish reusable 3D/scene assets, simulation backends, evaluation benchmarks, and both downloadable synthetic datasets and systems that generate such data.

3D Assets & Scene Datasets

Table: 3D assets and scene datasets

Year	Resource / Acronym	Reference	Publication / Type	Project	Official Code
2026	ManiTwin-100K	ManiTwin: Scaling Data-Generation-Ready Digital Object Dataset to 100K	arXiv
2026	PhysX-Mobility (dataset)	PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Image	CVPR
2026	PhysXVerse (dataset)	PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects	arXiv
2025	PhysXNet & PhysXNet-XL	PhysX-3D: Physical-Grounded 3D Asset Generation	NeurIPS
2024	HSSD-200	Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation	CVPR
2023	Objaverse	Objaverse: A Universe of Annotated 3D Objects	CVPR
2023	Objaverse-XL	Objaverse-XL: A Universe of 10M+ 3D Objects	NeurIPS
2023	OmniObject3D	OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation	CVPR
2023	ScanNet++	ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes	ICCV
2022	ABO	ABO: Dataset and Benchmarks for Real-World 3D Object Understanding	CVPR
2022	GSO	Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items	ICRA
2022	ObjectFolder 2.0	ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer	CVPR
2022	ProcTHOR	ProcTHOR: Large-Scale Embodied AI Using Procedural Generation	NeurIPS
2021	3D-FRONT	3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics	ICCV
2021	HM3D	Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI	NeurIPS
2019	Replica	The Replica Dataset: A Digital Replica of Indoor Spaces	arXiv
2017	Matterport3D	Matterport3D: Learning from RGB-D Data in Indoor Environments	3DV
2015	ShapeNet	ShapeNet: An Information-Rich 3D Model Repository	arXiv

Simulation & Rendering Backends

Table: Simulation and rendering backends

Year	Resource / Acronym	Reference	Publication / Type	Project	Official Code
2026	Genie Sim 3.0	Genie Sim 3.0: A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot	arXiv
2026	Tac2Real	Tac2Real: Reliable and GPU Visuotactile Simulation for Online Reinforcement Learning and Zero-Shot Real-World Deployment	arXiv
2026	UniVTAC	UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking	arXiv
2025	GenManip	GenManip: LLM-driven Simulation for Generalizable Instruction-Following Manipulation	CVPR
2025	NVIDIA Isaac Sim	Official software documentation	Software (open-source repository released in 2025)
2025	ManiSkill3	ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI	RSS
2025	MuJoCo Playground	MuJoCo Playground	arXiv
2025	Newton	Open-source GPU-accelerated physics engine for robotics	Software
2025	RoboVerse	RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning	arXiv
2025	Taccel	Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation	NeurIPS
2025	TacFlex	TacFlex: Multimode Tactile Imprints Simulation for Visuotactile Sensors With Coating Patterns	T-RO
2025	TacSL	TacSL: A Library for Visuotactile Sensor Simulation and Learning	T-RO
2024	DiffTactile	DiffTactile: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation	ICLR
2024	Genesis	Generative and universal physics engine for robotics and beyond	Software
2024	Habitat 3.0	Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots	ICLR
2024	TacEx	TacEx: GelSight Tactile Simulation in Isaac Sim -- Combining Soft-Body and Visuotactile Simulators	CoRL 2024 Workshop / arXiv
2023	Orbit	Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments	RA-L
2023	RoboHive	RoboHive: A Unified Framework for Robot Learning	NeurIPS
2023	Tacchi	Tacchi: A Pluggable and Low Computational Cost Elastomer Deformation Simulator for Optical Tactile Sensors	RA-L
2022	Tactile Gym 2.0	Tactile Gym 2.0: Sim-to-Real Deep Reinforcement Learning for Comparing Low-Cost High-Resolution Robot Touch	RA-L
2022	Tacto	TACTO: A Fast, Flexible, and Open-Source Simulator for High-Resolution Vision-Based Tactile Sensors	RA-L
2022	Taxim	Taxim: An Example-based Simulation Model for GelSight Tactile Sensors	RA-L
2021	Brax	Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation	NeurIPS
2021	Habitat 2.0	Habitat 2.0: Training Home Assistants to Rearrange their Habitat	NeurIPS
2021	Isaac Gym	Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning	NeurIPS
2016	PyBullet	PyBullet	Software (official citation: 2016--2021)
2021	ThreeDWorld (TDW)	ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation	NeurIPS
2020	SAPIEN	SAPIEN: A SimulAted Part-based Interactive ENvironment	CVPR
2019	PyRep	PyRep: Bringing V-REP to Deep Robot Learning	arXiv
2013	V-REP (now CoppeliaSim)	V-REP: A Versatile and Scalable Robot Simulation Framework	IROS
2012	MuJoCo	MuJoCo: A physics engine for model-based control	IROS

Benchmarks

Table: Simulation benchmarks

Year	Resource / Acronym	Reference	Publication / Type	Project	Official Code
2026	FlatLab	FlatLab: A Unified Methodology Framework and Simulation-Based Benchmark for Robotic Manipulation of Flat Objects	arXiv / simulation benchmark
2026	BiCoord	BiCoord: A Bimanual Manipulation Benchmark towards Long-Horizon Spatial-Temporal Coordination	arXiv
2026	DuoBench	DuoBench: A Reproducible Benchmark for Bimanual Manipulation in Simulation and the Real World	arXiv
2026	MIKASA	Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning	ICLR
2026	REALM	REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation	RA-L
2026	RMBench	RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design	arXiv
2026	RoboCasa365	RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots	ICLR
2026	RoboDojo	RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies	arXiv
2026	RoboMemArena	RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark	arXiv
2026	RoboMME	RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies	ICML
2026	RobotArena ∞	RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translation	ICLR
2025	ManiSkill-HAB	ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Tasks	ICLR
2025	RoboCerebra	RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation	NeurIPS
2025	RoboEval	RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation	arXiv
2025	RoboTwin 2.0	RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation	arXiv
2025	VLABench	VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks	ICCV
2024	BEHAVIOR-1K	BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation	arXiv
2024	BiGym	BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmark	CoRL
2024	GarmentLab	GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation	NeurIPS
2024	HumanoidBench	HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation	RSS
2024	ManiSkill-ViTac 2025	ManiSkill-ViTac 2025: Challenge on Manipulation Skill Learning With Vision and Tactile Sensing	arXiv
2024	RoboCasa	RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots	RSS
2024	RoboTwin	RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)	ECCV Workshop
2024	SIMPLER	Evaluating Real-World Robot Manipulation Policies in Simulation	CoRL
2024	THE COLOSSEUM	THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation	RSS
2023	ARNOLD	ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes	ICCV
2023	DaXBench	DaXBench: Benchmarking Deformable Object Manipulation with Differentiable Physics	ICLR
2023	DexArt	DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects	CVPR
2023	LIBERO	LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning	NeurIPS
2023	ManiSkill2	ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills	ICLR
2023	VIMA	VIMA: General Robot Manipulation with Multimodal Prompts	ICML
2022	Bi-DexHands	Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning	NeurIPS
2022	CALVIN	CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks	RA-L
2021	PlasticineLab	PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics	ICLR
2020	RLBench	RLBench: The Robot Learning Benchmark & Learning Environment	RA-L
2020	SoftGym	SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation	CoRL
2019	Meta-World	Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning	CoRL

Simulation/Synthetic Datasets & Data-Generation Systems

Table: Simulation/synthetic datasets and data-generation systems

Year	Resource / Acronym	Reference	Publication / Type	Project	Official Code
2026	FetchMan	FetchMan: Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences	arXiv / sim-to-real data-generation pipeline
2026	Self-Demonstrated Generative Control	Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation	arXiv / self-generated rollout data
2026	MolmoB0T data	MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation	arXiv / simulation dataset
2025	Dex1B	Dex1B: Learning with 1B Demonstrations for Dexterous Manipulation	RSS / dataset
2025	DexMimicGen	DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning	ICRA / data-generation system
2025	DreamGen	DreamGen: Unlocking Generalization in Robot Learning through Video World Models	CoRL / world-model-generated data
2025	GR00T-X-Embodiment-Sim	GR00T N1: An Open Foundation Model for Generalist Humanoid Robots	arXiv / simulation dataset
2025	InternData-A1	InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policy	arXiv / synthetic dataset
2025	SynGrasp-1B	GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data	arXiv / synthetic dataset
2024	GenSim	GenSim: Generating Robotic Simulation Tasks via Large Language Models	ICLR / task-generation system
2024	GenSim2	GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs	CoRL / data-generation framework
2024	RoboGen	RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation	ICML / generative simulation system
2023	DexGraspNet	DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation	ICRA / dataset
2023	MimicGen	MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations	CoRL / data-generation system
2021	ACRONYM	ACRONYM: A Large-Scale Grasp Dataset Based on Simulation	ICRA / dataset
2021	robomimic	What Matters in Learning from Offline Human Demonstrations for Robot Manipulation	CoRL / framework & simulated human demonstrations

World Models as Simulators

Table: World models as simulators

Year	Acronym	Paper	Venue	Project	Repo@GitHub
2026	ViTacWorld	ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation	arXiv
2026	N0-TWAM	$N_0$-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation	arXiv
2026	OVTF	Disentangling Visuo-Tactile Foresight: Oracle-Guided Interface Discovery for World Action Models	arXiv
2026		World Model for Robot Learning: A Comprehensive Survey	arXiv
2026	GigaBrain-0.5M*	GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learning	arXiv
2026	PlayWorld	PlayWorld: Learning Robot World Models from Autonomous Play	arXiv
2026	RehearseVLA	RehearseVLA: Simulated Post-Training for VLAs with Physically-Consistent World Model	CVPR
2026	RISE	RISE: Self-Improving Robot Policy with Compositional World Model	RSS
2026	WMPO	WMPO: World Model-based Policy Optimization for Vision-Language-Action Models	ICLR
2026	World-Gymnast	World-Gymnast: Training Robots with Reinforcement Learning in a World Model	arXiv
2025		A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models	arXiv
2025		Evaluating Gemini Robotics Policies in a Veo World Simulator	arXiv
2025	DiWA	DiWA: Diffusion Policy Adaptation with World Models	CoRL
2025	DreamGen	DreamGen: Unlocking Generalization in Robot Learning through Video World Models	CoRL
2025	GigaWorld-0	GigaWorld-0: World Models as Data Engine to Empower Embodied AI	arXiv
2025	ProphRL	Reinforcing Action Policies by Prophesying	arXiv
2025	VLA-RFT	VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators	arXiv
2025	World-Env	World-Env: Leveraging World Model as a Virtual Environment for VLA Post-Training	arXiv
2025	World4RL	World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation	arXiv
2025	WorldEval	WorldEval: World Model as Real-World Robot Policies Evaluator	arXiv
2025	WorldGym	WorldGym: World Model as An Environment for Policy Evaluation	arXiv
2024	3DFF	Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigation	CoRL
2024	UniSim	Learning Interactive Real-World Simulators	ICLR
2023	DayDreamer	DayDreamer: World Models for Physical Robot Learning	CoRL
2023	DreamerV3	Mastering Diverse Domains through World Models	arXiv
2020	Dreamer	Dream to Control: Learning Behaviors by Latent Imagination	ICLR
2019	PlaNet	Learning Latent Dynamics for Planning from Pixels	ICML
2018	World Models	Recurrent World Models Facilitate Policy Evolution	NeurIPS
2016		DeepMind Lab	arXiv

General Data

Visual Data

Table: Visual data

Year	Acronym	Paper	Venue	Project	Repo@GitHub
2025	FiRE	FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval	SIGIR
2025	RoboAfford++	RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation	arXiv
2024	LLaVA-OneVision-SI	LLaVA-OneVision: Easy Visual Task Transfer	TMLR
2024	LLaVA-Video-178K	LLaVA-Video-178K: Video Instruction Tuning With Synthetic Data	TMLR
2024	MultiUI	Harnessing Webpage UIs for Text-Rich Visual Understanding	ICLR
2024	PixMo	Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models	CVPR
2024	PixMo-Points	Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models	CVPR
2024	RoboPoint	RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics	CoRL
2023	PACO-LVIS	PACO: Parts and Attributes of Common Objects	CVPR
2023	SA-1B	Segment Anything (SA-1B dataset)	ICCV
2023	ShareGPT4V	ShareGPT4V: Improving Large Multi-Modal Models with Better Captions	ECCV
2022	ChartQA	ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning	ACL
2021	Ego4D NLQ	Ego4D: Around the World in 3,000 Hours of Egocentric Video	CVPR
2021	TextOCR	TextOCR: Towards Large-Scale End-to-End Reasoning for Arbitrary-Shaped Scene Text	CVPR
2020	DocVQA	DocVQA: A Dataset for VQA on Document Images	WACV
2017	Visual Genome	Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations	IJCV
2016	ADE20K	Semantic Understanding of Scenes through the ADE20K Dataset	IJCV
2016	RefCOCO	Modeling Context in Referring Expressions	ECCV
2015	VQA	VQA: Visual Question Answering	ICCV
2014	COCO	Microsoft COCO: Common Objects in Context	ECCV
2019	Objects365	Objects365: A Large-Scale, High-Quality Dataset for Object Detection	ICCV

Segmentation & Localization

Table: Segmentation and localization data

Year	Acronym	Paper	Venue	Project	Repo@GitHub
2025	PTG-FSCIR	Pseudo-triplet Guided Few-shot Composed Image Retrieval	IJCNN
2024	MMScan	MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations	NeurIPS
2023	VL-Grasp	VL-Grasp: a 6-Dof Interactive Grasp Policy for Language-Oriented Objects in Cluttered Indoor Scenes	IROS
2022	SQA3D	SQA3D: Situated Question Answering in 3D Scenes	ICLR
2022	VOST	Breaking the "Object" in Video Object Segmentation	CVPR
2021	ARKitScenes	ARKitScenes: A Diverse Real-World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data	NeurIPS
2021	ScanQA	ScanQA: 3D Question Answering for Spatial Scene Understanding	CVPR
2020	3DSSG	Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions	CVPR
2019	3RScan	RIO: 3D Object Instance Re-Localization in Changing Indoor Environments	ICCV
2019	ALFRED	ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks	CVPR
2019	CLEVRER	CLEVRER: Collision Events for Video Representation and Reasoning	ICLR
2019	LVIS	LVIS: A Dataset for Large Vocabulary Instance Segmentation	CVPR
2016	COLMAP	Structure-from-Motion Revisited	CVPR
2015	ORB-SLAM	ORB-SLAM: A Versatile and Accurate Monocular SLAM System	T-RO
2012		Indoor Segmentation and Support Inference from RGBD Images	ECCV

Video & Temporal Data

Table: Video and temporal data

Year	Acronym	Paper	Venue	Project	Repo@GitHub
2026	RoVid-X	Rethinking Video Generation Model for the Embodied World	ICML
2026	D2E	D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI	ICLR
2025	PLAICraft	PLAICraft: Large-Scale Time-Aligned Vision-Speech-Action Dataset for Embodied AI	arXiv
2024	Moment-10M	Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning	ICML
2024	RH20T-P	RH20T-P: A Primitive-Level Robotic Manipulation Dataset Towards Composable Generalization Agents in Real-world Scenarios	IROS
2023	HiREST	Hierarchical Video-Moment Retrieval and Step-Captioning	CVPR
2023	RoboFail	REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction (RoboFail dataset)	CoRL
2023	RoboVQA	RoboVQA: Multimodal Long-Horizon Reasoning for Robotics	ICRA
2019	COIN	COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis	CVPR
2019	RoboNet	RoboNet: Large-Scale Multi-Robot Learning	CoRL
2018	IntPhys	IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning	TPAMI
2017	Charades-STA	TALL: Temporal Activity Localization via Language Query (Charades-STA dataset)	ICCV
2017	DiDeMo	Localizing Moments in Video with Natural Language	ICCV
2017	ScanNet	ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes	CVPR
2011	Cornell Grasp Dataset	Efficient Grasping from RGBD Images: Learning using a New Rectangle Representation	ICRA
2022	InfLevel	Benchmarking Progress to Infant-Level Physical Reasoning in AI	TMLR
2019	OCR-VQA	OCR-VQA: Visual Question Answering by Reading Text in Images	ICDAR
2025	RobAVA	RobAVA: A Large-scale Dataset and Baseline Towards Video-based Robotic Arm Action Understanding	ICCV

Planning / Task Decomposition

Table: Planning and task decomposition data

Year	Acronym	Paper	Venue	Project	Repo@GitHub
2023	EgoPlan-IT	EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning	IJCV
2023	EmbodiedGPT	EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought	NeurIPS
2023	LLaRP	Large Language Models as Generalizable Policies for Embodied Tasks	ICLR
2022	EgoTaskQA	EgoTaskQA: Understanding Human Tasks in Egocentric Videos	NeurIPS
2025	WAP	World-aware Planning Narratives Enhance Large Vision-Language Model Planner	NeurIPS

Grasp Data

Table: Grasp data

Year	Acronym	Paper	Venue	Project	Repo@GitHub
2026	MapleGrasp	MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping	WACV
2024		Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance	ECCV
2024	DexGraspNet 2.0	DexGraspNet 2.0: Learning Generative Dexterous Grasping in Large-scale Synthetic Cluttered Scenes	CoRL
2024	Grasp-Anything	Grasp-Anything: Large-scale Grasp Dataset from Foundation Models	ICRA
2021	SuctionNet-1Billion	SuctionNet-1Billion: A Large-Scale Benchmark for Suction Grasping	RAL
2018	Jacquard	Jacquard: A Large Scale Dataset for Robotic Grasp Detection	IROS
2020	GraspNet-1Billion	GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping	CVPR

Embodied Foundation Models

Representative VLA and WAM Methods

Representative VLA and WAM methods by release time, model type, institution, and data source.

Data sources: = real-robot data; = UMI data; = egocentric data; = simulation data; = general data.

Table: Representative VLA and WAM methods (74 entries)

Time	Method	Institution	Project	Model	Data
2023.3	PaLM-E	Google		VLA
2023.7	RT-2	DeepMind		VLA
2024.5	Octo	UC Berkeley		VLA
2024.6	OpenVLA	Stanford University		VLA
2024.10	GR-2	ByteDance		VLA
2024.10	π0	Physical Intelligence		VLA
2024.10	RDT-1B	THU		VLA
2024.11	CogACT	THU		VLA
2025.1	SpatialVLA	Shanghai AI Lab		VLA
2025.1	UP-VLA	THU		VLA
2025.3	HybridVLA	PKU		VLA
2025.3	GR00T N1	NVIDIA		VLA
2025.3	GO-1	Shanghai AI Lab & AgiBot		VLA
2025.3	CoT-VLA	NVIDIA		VLA
2025.4	UWM	University of Washington		WAM
2025.4	π0.5	Physical Intelligence		VLA
2025.5	UniVLA	HKU		VLA
2025.6	SmolVLA	Hugging Face		VLA
2025.6	GR00T N1.5	NVIDIA		VLA
2025.7	DreamVLA	SJTU		VLA
2025.7	EgoVLA	UC San Diego		VLA
2025.7	Being-H0	PKU		VLA
2025.7	GR-3	ByteDance Seed		VLA
2025.7	H-RDT	THU		VLA
2025.8	GalaxeaVLA(G0)	Galaxea		VLA
2025.8	ReconVLA	HKUST		VLA
2025.8	MolmoAct	AI2 & UW		VLA
2025.8	EO-1	Shanghai AI Lab		VLA
2025.9	RynnVLA-001	DAMO		VLA
2025.10	X-VLA	THU		VLA
2025.10	InternVLA-M1	Shanghai AI Lab		VLA
2025.10	VITRA	THU		VLA
2025.11	METIS	PKU		VLA
2025.11	iFlyBot-VLA	iFLYTEK & LindenBot		VLA
2025.12	VideoVLA	XJTU		VLA
2025.12	Motus	THU		WAM
2025.12	GR00T N1.6	NVIDIA		VLA
2026.1	InternVLA-A1	Shanghai AI Lab		VLA
2026.1	LaST_0	PKU		VLA
2026.1	Being-H0.5	BeingBeyond		VLA
2026.1	LingbotVLA	Ant Group		VLA
2026.1	LingbotVA	Ant Group		WAM
2026.2	RDT2	THU		VLA
2026.2	Xiaomi-Robotics-0	Xiaomi		VLA
2026.2	ABot-M0	AMAP CV Lab		VLA
2026.2	DM0	Dexmal		VLA
2026.2	LDA-1B	PKU		WAM
2026.2	DreamZero	NVIDIA		WAM
2026.2	EgoScale	NVIDIA		VLA
2026.3	GigaWorld-Policy	GigaAI		WAM
2026.3	MolmoB0T	AI2		VLA
2026.3	UniDex	THU		VLA
2026.4	JoyAI-RA 0.1	JD		VLA
2026.4	π0.7	Physical Intelligence		VLA
2026.4	GR00T N1.7	NVIDIA		VLA
2026.4	Being-H0.7	BeingBeyond		WAM
2026.4	MotuBrain	Shengshu		WAM
2026.5	Qwen-VLA	Qwen		VLA
2026.5	Wall-OSS-0.5	X Square Robot		VLA
2026.5	MolmoAct2	UW & AI2		VLA
2026.6	Wall-WM	X Square Robot		WAM
2026.6	LaST-HD	PKU		VLA
2026.6	Hy-Embodied-0.5-VLA	Tencent Robotics X		VLA
2026.6	Kairos	ACE Robotics		WAM
2026.6	Qwen-RobotManip	Qwen		VLA
2026.7	ABot-M0.5	AMAP CV Lab		WAM
2026.7	ACE-Brain-0.5	ACE-Robotics		VLA
2026.7	InternVLA-A1.5	Shanghai AI Lab		VLA
2026.7	LingbotVLA 2.0	Ant Group		VLA
2026.7	LingbotVA 2.0	Ant Group		WAM
2026.7	HumanScale	PKU		WAM
2026.7	GigaWorld-Policy-0.5	GigaAI		WAM
2026.7	Xiaomi-Robotics-1	Xiaomi		VLA
2026.7	Xiaomi-Robotics-U0	Xiaomi Robotics		WAM

---

Contributing

Contributions are welcome! Please open an issue or pull request to add resources, fix links, update metadata, or suggest improvements. For new resources, please include the paper title, project page, code link if available, release date, scale, and the most relevant data category.

---

Acknowledgement

If you find this repository useful, please consider citing our paper:

@misc{ye2026datapyramid,
      title={{Data Pyramid for Embodied Manipulation}},
      author={Yifan Ye and Yankai Fu and Yaoxu Lv and Bohan Hou and Jun Cen and Lingdong Kong and Duo Zheng and Tianxing Chen and Jiaming Liu and Ziang Cao and Yunfan Lou and Wei Chow and Xian Sun and Yingshuo Wang and Kuangzhi Ge and Xiaowei Chi and Xidong Zhang and Zhibo Pang and Yiwu Zhong and Sirui Han and Zhihe Lu and Weihao Yuan and Qifeng Chen and Michael Yu Wang and Yao Mu and Ziwei Liu and Jianfei Yang and Ping Luo and Shanghang Zhang},
      year={2026},
      eprint={2607.24744},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.24744}
}