Awesome-Robot-Learning-from-Human-Videos

June 22, 2026 ยท View on GitHub

Awesome Maintenance PR's Welcome arXiv

teaser

๐Ÿ“– Overview

This repository provides a curated reading list for robot learning from human videos (LfHV), with an emphasis on human-robot skill transfer techniques in task, observation, and action levels. It also covers representative works on human-object interaction analysis and human video sources that are widely used in the literature.

Papers with publicly released code are marked with a star ๐ŸŒŸ. Papers with real-world robot experiments are marked with a robot ๐Ÿค–.

Contributions are welcome. If you find missing papers or inaccurate classifications, feel free to open a pull request or contact me via email.

The related survey paper can be found at this link. In this survey, you can find answers to the following interesting questions:

๐Ÿ” How much more efficient is human video collection compared to robot teleoperation?

๐Ÿ” What types of information can be transferred from human videos to robot manipulation?

๐Ÿ” Which is more widely adopted in LfHV, egocentric or exocentric video data?

๐Ÿ” How do imitation learning and reinforcement learning incorporate information from human videos?

๐Ÿ” How can you design your own LfHV framework for specific application needs?

๐Ÿ” How have open-source human video datasets evolved over time?

๐Ÿ” What are the most promising future research directions in LfHV?

๐Ÿ” ...

We hope that the answers presented in this survey can inspire future research and contribute to the development of generalist robotic systems. A more concise version of this survey is coming soon.

We sincerely appreciate these blogs for helping distill and promote our work:

๐Ÿ“Œ Citation

If you find this work helpful for your research, please kindly consider citing our paper:

@article{ma2026robot,
  title={Robot Learning from Human Videos: A Survey},
  author={Ma, Junyi and Zhang, Erhang and Yang, Haoran and Li, Ditao and Xu, Chenyang and Wang, Guangming and Wang, Hesheng},
  journal={arXiv preprint arXiv:2604.27621},
  year={2026}
}

๐Ÿ—‚๏ธ Table of Contents

taxonomy

๐Ÿ”„ Human-Robot Skill Transfer

Task-Oriented Transfer

This section covers methods that infer task structures and intents from human videos to guide robot decision-making at the task level.

2026

  • ๐Ÿค– [arXiv 2026.02] Act, Sense, Act: Learning Non-Markovian Active Perception Strategies from Large-Scale Egocentric Human Data [Code] [Page] [Task intents]

2025

  • ๐Ÿค– [IJRR] FMimic: Foundation Models Are Fine-Grained Action Learners from Human Videos [Code] [Page] [Task structures]
  • ๐Ÿค– [ICRA] Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models [Code] [Page] [Task structures]
  • ๐Ÿค– [CoRL] Action-Free Reasoning for Policy Generalization [Code] [Page] [Task structures]
  • [ICASSP] Interactive Robot Action Replanning Using Multimodal LLM Trained from Human Demonstration Videos [Code] [[Page]] [Task structures]
  • ๐Ÿค– [arXiv 2025.12] PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence [Code] [Page] [Task structures]
  • [arXiv 2025.11] Robot Confirmation Generation and Action Planning Using Long-Context Q-Former Integrated with Multimodal LLM [Code] [[Page]] [Task structures]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.09] From Watch to Imagine: Steering Long-Horizon Manipulation via Human Demonstration and Future Envisionment [Code] [Page] [Task structures]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.08] EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videos [Code] [[Page]] [Task structures]

2024

  • ๐ŸŒŸ๐Ÿค– [RA-L] GPT-4V(ision) for Robotics: Multimodal Task Planning From Human Demonstration [Code] [Page] [Task structures]
  • ๐Ÿค– [NeurIPS] VLMimic: Vision Language Models Are Visual Imitation Learner for Fine-Grained Actions [Code] [Page] [Task structures]
  • ๐Ÿค– [RSS] Vid2Robot: End-to-End Video-Conditioned Policy Learning with Cross-Attention Transformers [Code] [Page] [Task intents]
  • ๐ŸŒŸ๐Ÿค– [IROS] Knowledge-Based Programming by Demonstration Using Semantic Action Models for Industrial Assembly [Code] [Page] [Task structures]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2024.10] VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Model [Code] [Page] [Task structures]

2023

  • ๐Ÿค– [T-SMC] Watch and Act: Learning Robotic Manipulation From Visual Demonstration [Code] [Page] [Task structures]
  • ๐ŸŒŸ๐Ÿค– [CoRL] XSkill: Cross Embodiment Skill Discovery [Code] [Page] [Task intents]
  • ๐Ÿค– [arXiv 2023.12] Learning Multi-Step Manipulation Tasks from A Single Human Demonstration [Code] [[Page]] [Task structures]

2022

  • ๐Ÿค– [T-MECH] Explicit-to-Implicit Robot Imitation Learning by Exploring Visual Content Change [Code] [Page] [Task structures]
  • ๐Ÿค– [ICRA] Learning Periodic Tasks from Human Demonstrations [Code] [Page] [Task structures]
  • ๐Ÿค– [CoRL] BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning [Code] [Page] [Task intents]
  • ๐ŸŒŸ [CoRL] Cross-Domain Transfer via Semantic Skill Imitation [Code] [Page] [Task structures]

2019

  • ๐ŸŒŸ๐Ÿค– [NeurIPS] Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller [Code] [Page] [Task intents]
  • ๐Ÿค– [IROS] Learning Actions from Human Demonstration Video for Robotic Manipulation [Code] [[Page]] [Task structures]

2018

  • ๐Ÿค– [RSS] One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning [Code] [[Page]] [Task intents]
  • ๐ŸŒŸ๐Ÿค– [ICRA] Translating Videos to Commands for Robotic Manipulation with Deep Recurrent Neural Networks [Code] [[Page]] [Task structures]
  • ๐Ÿค– [arXiv 2018.10] One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks [Code] [Page] [Task intents]

2017

  • ๐ŸŒŸ๐Ÿค– [RSS] Unsupervised Perceptual Rewards for Imitation Learning [Code] [Page] [Task intents]
  • [RA-L] Unsupervised Linking of Visual Features to Textual Descriptions in Long Manipulation Activities [Code] [[Page]] [Task structures]

2015

  • [AAAI] Robot Learning Manipulation Action Plans by "Watching" Unconstrained Videos from the World Wide Web [Code] [[Page]] [Task structures]

Observation-Oriented Transfer

This section focuses on bridging the observation gap between humans and robots via transformed videos and visual embeddings.

2026

  • ๐Ÿค– [Science Robotics] Visual-Tactile Pretraining and Online Multitask Learning for Humanlike Manipulation Dexterity [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [ICRA] Masquerade: Learning from In-the-Wild Human Videos Using Data-Editing [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ๐Ÿค– [ICRA] MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos [Code] [Page] [Visual embeddings]
  • ๐Ÿค– [AAAI] Human2Robot: Learning Robot Actions from Paired Human-Robot Videos [Code] [Page] [Transformed videos]
  • ๐Ÿค– [arXiv 2026.04] WARPED: Wrist-Aligned Rendering for Robot Policy Learning from Egocentric Human Demonstrations [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2026.02] FRAPPE: Infusing World Modeling into Generalist Policies via Multiple Future Representation Alignment[Code] [Page] [Visual embeddings, VLA]

2025

  • ๐Ÿค– [NeurIPS] EgoBridge: Domain Adaptation for Generalizable Imitation from Egocentric Human Data [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [CVPR] Mitigating the Human-Robot Domain Discrepancy in Visual Pre-Training for Robotic Manipulation [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [RA-L] GR-MG: Leveraging Partially-Annotated Data via Multi-Modal Goal-Conditioned Policy [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [ICRA] One-Shot Imitation Under Mismatched Execution [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ [IROS] Ag2x2: Robust Agent-Agnostic Visual Representations for Zero-Shot Bimanual Manipulation [Code] [Page] [Transformed videos]
  • ๐Ÿค– [IROS] VTAO-BiManip: Masked Visual-Tactile-Action Pre-Training with Object Understanding for Bimanual Dexterous Manipulation [Code] [Page] [Visual embeddings]
  • ๐Ÿค– [CoRL] Generative Visual Foresight Meets Task-Agnostic Pose Estimation in Robotic Table-Top Manipulation [Code1] [Code2] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [CoRL] ImMimic: Cross-Domain Imitation from Human Videos via Mapping and Interpolation [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [CoRL] Phantom: Training Robots Without Robots Using Only Human Videos [Code] [Page] [Transformed videos]
  • ๐Ÿค– [CoRL] Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ [arXiv 2025.12] H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ [arXiv 2025.12] Mitty: Diffusion-Based Human-to-Robot Video Generation [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ [arXiv 2025.12] X-Humanoid: Robotize Human Videos to Generate Humanoid Videos at Scale [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.12] mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.12] World Models Can Leverage Human Videos for Dexterous Manipulation [Code] [Page] [Visual embeddings]
  • ๐Ÿค– [arXiv 2025.10] Trajectory-Conditioned Cross-Embodiment Skill Transfer [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.09] MimicDreamer: Aligning Human and Robot Demonstrations for Scalable VLA Training [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.09] RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulation [Code] [Page] [Visual embeddings]
  • ๐Ÿค– [CoRL] FLARE: Robot Learning with Implicit World Modeling [Code] [Page] [Visual embeddings, VLA]
  • ๐Ÿค– [arXiv 2025.05] H2R: A Human-to-Robot Data Augmentation for Robot Pre-Training from Videos [Code] [Page] [Transformed videos]

2024

  • ๐Ÿค– [T-ASE] Contrast, Imitate, Adapt: Learning Robotic Skills from Raw Human Videos [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [ICLR] Unleashing Large-Scale Video Generative Pre-Training for Visual Robot Manipulation [Code] [Page] [Visual embeddings]
  • ๐Ÿค– [RSS] Vid2Robot: End-to-End Video-Conditioned Policy Learning with Cross-Attention Transformers [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [RSS] Learning Manipulation by Predicting Interaction [Code] [Page] [Visual embeddings]
  • [ICRA] Masked Visual-Tactile Pre-Training for Robot Manipulation [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [ICRA] Robotic Offline RL from Internet Videos via Value-Function Pre-Training [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [IROS] Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ๐Ÿค– [CoRL] What Makes Pre-Trained Visual Representations Successful for Robust Manipulation? [Code] [Page] [Visual embeddings]
  • ๐Ÿค– [arXiv 2024.10] GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation [Code] [Page] [Visual embeddings]

2023

  • ๐ŸŒŸ๐Ÿค– [ICML] LIV: Language-Image Representations and Rewards for Robotic Control [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [ICLR] VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [NeurIPS] Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ๐Ÿค– [NeurIPS] Where Are We in the Search for an Artificial Visual Cortex for Embodied Intelligence? [Code] [Page] [Visual embeddings]
  • ๐Ÿค– [RSS] Structured World Models from Human Videos [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [RSS] Language-Driven Representation Learning for Robotics [Code] [Page] [Visual embeddings]
  • [ICRA] Learning Video-Conditioned Policies for Unseen Manipulation Tasks [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [CoRL] AR2-D2: Training a Robot Without a Robot [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ๐Ÿค– [CoRL] An Unbiased Look at Datasets for Visuo-Motor Pre-Training [Code] [Page] [Visual embeddings]
  • ๐Ÿค– [arXiv 2023.04] Efficient Robot Skill Learning with Imitation from a Single Video for Contact-Rich Fabric Manipulation [Code] [Page] [Visual embeddings]

2022

  • ๐ŸŒŸ๐Ÿค– [CoRL] R3M: A Universal Visual Representation for Robot Manipulation [Code] [Page] [Visual embeddings]
  • ๐Ÿค– [RSS] Human-to-Robot Imitation in the Wild [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ๐Ÿค– [CoRL] Real-World Robot Learning with Masked Visual Pre-Training [Code] [Page] [Visual embeddings]
  • ๐Ÿค– [CoRL] RoboTube: Learning Household Manipulation from Human Videos with Simulated Twin Environments [Code] [Page] [Visual embeddings]
  • [Machines] Learning by Watching via Keypoint Extraction and Imitation Learning [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ [arXiv 2022.03] Masked Visual Pre-Training for Motor Control [Code] [Page] [Visual embeddings]

2021

  • ๐ŸŒŸ๐Ÿค– [RSS] Learning Generalizable Robotic Reward Functions from "In-the-Wild" Human Videos [Code] [Page] [Visual embeddings]
  • [IROS] Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ [CoRL] XIRL: Cross-Embodiment Inverse Reinforcement Learning [Code] [Page] [Visual embeddings]
  • ๐Ÿค– [arXiv 2021.03] Manipulator-Independent Representations for Visual Imitation [Code] [Page] [Transformed videos]

2020

  • ๐Ÿค– [RSS] AVID: Learning Multi-Stage Tasks Via Pixel-Level Translation of Human Videos [Code] [Page] [Transformed videos]
  • ๐ŸŒŸ๐Ÿค– [CoRL] Reinforcement Learning with Videos: Combining Offline Observations with Interaction [Code] [Page] [Visual embeddings]

2018

  • ๐ŸŒŸ๐Ÿค– [RA-L] Deep Episodic Memory: Encoding, Recalling, and Predicting Episodic Experiences for Robot Action Execution [Code] [Page] [Visual embeddings]
  • ๐ŸŒŸ๐Ÿค– [ICRA] Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation [Code] [Page] [Visual embeddings]

2017

  • ๐ŸŒŸ๐Ÿค– [CVPRW] Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation [Code] [Page] [Visual embeddings]

Action-Oriented Transfer

This section includes methods that transfer actionable motion knowledge from human videos to robot control, including affordances (explicit interaction info) and latent actions.

2026

  • ๐ŸŒŸ๐Ÿค– [TRO&IROS] AgiBot World Colosseo: A Large-Scale Manipulation Platform for Scalable and Intelligent Embodied Systems [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [ICLR&NeurIPS Workshop] ViPRA: Video Prediction for Robot Actions [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [CVPR] Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wild [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [CVPR] UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos [Code] [Page][Affordances]
  • ๐ŸŒŸ๐Ÿค– [CVPR] Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [AAAI] H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ [AAAI] Towards Affordance-Aware Robotic Dexterous Grasping with Human-Like Priors [Code] [Page] [Affordances]
  • [ICASSP] CARE: Multi-Task Pretraining for Latent Continuous Action Representation in Robot Control [Code] [Page] [Latent actions]
  • ๐Ÿค– [RA-L] EMMA: Scaling Mobile Manipulation via Egocentric Human Data [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] DemoDiffusion: One-Shot Human Imitation Using Pre-Trained Diffusion Policy [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] Actron3D: Learning Actionable Neural Functions from Videos for Transferable Robotic Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2026.06] Unifying Egocentric Human and Robotic Data for VLA Pretraining [Code] [Page] [VLA]
  • ๐Ÿค– [arXiv 2026.06] Ego-Pi: VLA Fine-Tuning for Ego-Centric Human and Robot Data [Code] [Page] [VLA]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2026.05] HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2026.04] ActiveGlasses: Learning Manipulation with Active Vision from Ego-centric Human Demonstration [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2026.04] Learning Human-Intention Priors from Large-Scale Human Demonstrations for Robotic Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2026.04] Disentangled Robot Learning via Separate Forward and Inverse Dynamics Pretraining [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2026.03] PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2026.02] EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data [Code] [Page] [Affordances]
  • [arXiv 2026.02] MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2026.02] DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos [Code] [Page] [Latent actions]
  • ๐ŸŒŸ [CVPR] VideoWorld 2: Learning Transferable Knowledge from Real-world Videos [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2026.02] UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2026.02] VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model [Code] [Page] [Latent actions]
  • ๐Ÿค– [arXiv 2026.02] Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2026.02] ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2026.02] LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestion [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2026.01] Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization [Code] [Page][Affordances]
  • [arXiv 2026.01] Learning Latent Action World Models in the Wild [Code] [Page] [Latent actions]
  • [arXiv 2026.01] ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2026.01] CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos [Code] [Page] [Latent actions]

2025

  • ๐ŸŒŸ๐Ÿค– [CVPR] ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CVPR] Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioning [Code] [Page] [Affordances]
  • ๐Ÿค– [CVPR] GraphMimic: Graph-to-Graphs Generative Modeling from Videos for Policy Learning [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CVPR] VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICCV] AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICCV] DexH2R: A Benchmark for Dynamic Dexterous Grasping in Human-to-Robot Handover [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICCV] Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [ICCV&CoRL Workshop] 2HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [RSS] UniVLA: Learning to Act Anywhere with Task-Centric Latent Actions [Code] [Page] [Latent actions]
  • ๐Ÿค– [RSS] You Only Teach Once: Learn One-Shot Bimanual Robotic Manipulation from Video Demonstrations [Code] [Page] [Affordances]
  • ๐Ÿค– [RSS Workshop] ViSA-Flow: Accelerating Robot Skill Learning via Large-Scale Video Semantic Action Flow [Code] [Page] [Affordances]
  • [RSS Workshop] Learning from Watching: Scalable Extraction of Manipulation Trajectories from Human Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] ViViDex: Learning Vision-Based Dexterous Manipulation from Human Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] R+X: Retrieval and Execution from Everyday Human Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] SPOT: SE(3) Pose Trajectory Diffusion for Object-Centric Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] EgoMimic: Scaling Imitation Learning via Egocentric Video [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] Hand-Object Interaction Pretraining from Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] Hand-Object Interaction Pretraining from Videos [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [CoRL] Generalist Robot Manipulation beyond Action Labeled Data [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] In-N-On: Scaling Egocentric Manipulation with In-the-Wild and On-Task Data [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] Humanoid Policy ~ Human Policy [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL Using One Human Demonstration [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] FUNCTO: Function-Centric One-Shot Imitation Learning for Tool Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] MimicFunc: Imitating Tool Manipulation from a Single Human Video via Functional Correspondence [Code] [Page] [Affordances]
  • ๐ŸŒŸ [CoRL] Articulated Object Estimation in the Wild [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [CoRL] General Flow as Foundation Affordance for Scalable Robot Learning [Code] [Page] [Affordances]
  • ๐Ÿค– [CoRL Workshop] HERMES: Human-to-Robot Embodied Learning from Multi-Source Motion Data for Mobile Dexterous Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [Autonomous Robots] View: Visual Imitation Learning with Waypoints [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2025.12] GR-Dexter Technical Report [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2025.12] Emergence of Human to Robot Transfer in Vision-Language-Action Models [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.12] Motus: A Unified Latent Action World Model [Code] [Page] [Latent actions]
  • ๐Ÿค– [arXiv 2025.12] SurgWorld: Learning Surgical Robot Policies from Videos via World Modeling [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.11] Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.11] Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2025.11] LatBot: Distilling Universal Latent Actions for Vision-Language-Action Models [Code] [Page] [Latent actions]
  • ๐Ÿค– [arXiv 2025.10] Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ [arXiv 2025.10] DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videos [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2025.10] From Human Hands to Robot Arms: Manipulation Skills Transfer via Trajectory Alignment [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.10] Traj2Action: A Co-Denoising Framework for Trajectory-Guided Human-to-Robot Skill Transfer [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2025.09] Developing Vision-Language-Action Model from Egocentric Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.09] MotionTrans: Human VR Data Enable Motion-Level Learning for Robotic Manipulation Policies [Code] [Page] [Affordances]
  • ๐ŸŒŸ [arXiv 2025.08] Deep Sensorimotor Control by Imitating Predictive Models of Human Motion [Code] [Page] [Affordances]
  • ๐ŸŒŸ [arXiv 2025.07] EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.07] Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2025.07] GR-3 Technical Report [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.07] villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models [Code] [Page] [Latent actions]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.06] Object-Centric 3D Motion Field for Robot Learning from Human Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.05] EgoZero: Robot Learning from Smart Glasses [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2025.05] Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactions [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2025.04] Slot-Level Robotic Placement via Visual Imitation from Single Human Video [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.03] GR00T N1: An Open Foundation Model for Generalist Humanoid Robots [Code] [Page] [Affordances]
  • ๐Ÿค– [arXiv 2025.02] Video2Policy: Scaling Up Manipulation Tasks in Simulation Through Internet Videos [Code] [Page] [Affordances]

2024

  • ๐ŸŒŸ๐Ÿค– [ICLR] Learning to Act from Actionless Videos through Dense Correspondences [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICLR] Latent Action Pretraining from Videos [Code] [Page] [Latent actions]
  • ๐Ÿค– [ICLR] RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ECCV] Track2Act: Predicting Point Tracks from Internet Videos Enables Generalizable Robot Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ECCV] Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [RSS] HRP: Human Affordances for Robotic Pre-Training [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [RSS] ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection [Code] [Page] [Affordances]
  • ๐Ÿค– [ICRA] Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plan [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [IROS] Ditto: Demonstration Imitation by Trajectory Transformation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] Im2Flow2Act: Flow as the Cross-Domain Manipulation Interface [Code] [Page] [Affordances]
  • [arXiv 2024.11] IGOR: Image-Goal Representations Are the Atomic Control Units for Foundation Models in Embodied AI [Code] [Page] [Latent actions]
  • ๐Ÿค– [arXiv 2024.05] Vision-Based Manipulation from Single Human Video with Open-World Object Graphs [Code] [Page] [Affordances]

2023

  • ๐ŸŒŸ๐Ÿค– [TRO] K-VIL: Keypoints-Based Visual Imitation Learning [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [ICRA] Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CVPR] Affordances from Human Videos as a Versatile Representation for Robotics [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] MimicPlay: Long-Horizon Imitation Learning by Watching Human Play [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] DEFT: Dexterous Fine-Tuning for Hand Policies [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [RSS] Any-Point Trajectory Modeling for Policy Learning [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [RA-L&IROS] Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations [Code] [Page] [Affordances]
  • ๐Ÿค– [ICRA Workshop] Zero-Shot Robot Manipulation from Passive Human Videos [Code] [Page] [Affordances]
  • ๐Ÿค– [Sensors] Robot Programming from a Single Demonstration for High-Precision Industrial Insertion [Code] [Page] [Affordances]

2022

  • [RSS] You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration [Code] [Page] [Affordances]
  • ๐Ÿค– [CoRL] VideoDex: Learning Dexterity from Internet Videos [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [CoRL] Graph Inverse Reinforcement Learning from Diverse Videos [Code] [Affordances]
  • ๐Ÿค– [RSS] Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on YouTube [Code] [Page] [Affordances]
  • ๐ŸŒŸ๐Ÿค– [RA-L&IROS] From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation [Code] [Page] [Affordances]
  • ๐Ÿค– [RSS] Human-to-Robot Imitation in the Wild [Code] [Page] [Affordances]
  • ๐ŸŒŸ [ECCV] DexMV: Imitation Learning for Dexterous Manipulation from Human Videos [Code] [Page] [Affordances]
  • [arXiv 2022.11] Learning to Imitate Object Interactions from Internet Videos [Code] [Page] [Affordances]

2021

  • [CoRL] DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video [Code] [Page] [Affordances]

2020

  • ๐Ÿค– [CoRL] Model-Based Inverse Reinforcement Learning from Visual Demonstrations [Code] [Page] [Affordances]

2019

  • ๐Ÿค– [CoRL] Graph-Structured Visual Imitation [Code] [Page] [Affordances]

2017

  • ๐Ÿค– [CVPR] Learning Robot Activities from First-Person Human Videos Using Convolutional Future Regression [Code] [Page] [Affordances]

๐Ÿงฉ Data Foundations

Open-Source Datasets

This section collects open-source human video datasets.

DatasetYearVenueWebsiteViewpointOrganization (first author)
HumanNet2026arXiv 2026.05[Code] [Page]Ego+ExoPeking University
EgoLive2026arXiv 2026.04[Code] [Page]EgoJD
DreamDojo-HV2026arXiv 2026.02[Code] [Page]EgoNVIDIA
UniHand-Mix2026arXiv 2026.02[Code] [Page]EgoBeingBeyond
UniHand-2.02026arXiv 2026.01[Code] [Page]EgoBeingBeyond
Action100M2026arXiv 2026.01[Code] [Page]Ego+ExoMeta
EgoVid-5M2025NeurIPS[Code] [Page]EgoAlibaba Group
HO-Cap2025NeurIPS[Code] [Page]Ego+ExoUniversity of Texas at Dallas
IndEgo2025NeurIPS[Code] [Page]Ego+ExoFraunhofer IPK
HD-EPIC2025CVPR[Code] [Page]EgoUniversity of Bristol
HOT3D2025CVPR[Code] [Page]EgoMeta
TASTE-Rob2025CVPR[Code] [Page]EgoCUHK-Shenzhen
LVP-1M2025arXiv 2025.12[Code] [Page]Ego+ExoMIT
UniHand-1.02025arXiv 2025.07[Code] [Page]EgoBeingBeyond
EgoDex2025arXiv 2025.05[Code] [Page]EgoApple
PHยฒD2025arXiv 2025.03[Code] [Page]EgoUC San Diego
Egocentric-100k2025---[Code] [Page]EgoBuild AI
Egocentric-10k2025---[Code] [Page]EgoBuild AI
OakInk22024NeurIPS[Code] [Page]Ego+ExoShanghai Jiao Tong University
CaptainCook4D2024NeurIPS[Code] [Page]EgoUniversity of Texas at Dallas
Panda-70M2024CVPR[Code] [Page]Ego+ExoSnap Inc.
TACO2024CVPR[Code] [Page]Ego+ExoTsinghua University
Ego-Exo4D2024CVPR[Code] [Page]Ego+ExoMeta
Nymeria2024ECCV[Code] [Page]Ego+ExoMeta
ARCTIC2023CVPR[Code] [Page]Ego+ExoETH Zรผrich
HoloAssist2023ICCV[Code] [Page]EgoMicrosoft
RH20T-Human2023ICRA[Code] [Page]Ego+ExoShanghai Jiao Tong University
EPIC-KITCHENS-1002022IJCV[Code] [Page]EgoUniversity of Bristol
Assembly1012022CVPR[Code] [Page]Ego+ExoMeta
Ego4D2022CVPR[Code] [Page]EgoMeta
OakInk2022CVPR[Code] [Page]ExoShanghai Jiao Tong University
HOI4D2022CVPR[Code] [Page]EgoTsinghua University
EgoPAT3D2022CVPR[Code] [Page]EgoNew York University
AGD20K2022CVPR[Code] [Page]Ego+ExoUniversity of Science and Technology of China
EgoHOS2022ECCV[Code] [Page]EgoUniversity of Pennsylvania
DexYCB2021CVPR[Code] [Page]ExoNVIDIA
H2O2021ICCV[Code] [Page]EgoETH Zurich
MOW2021ICCV[Code] [Page]ExoUC Berkeley
100DOH2020CVPR[Code] [Page]Ego+ExoUniversity of Michigan
Kinetics-7002020arXiv 2020.10[Code] [Page]ExoGoogle
HowTo100M2019ICCV[Code] [Page]Ego+ExoENS
FreiHAND2019ICCV[Code] [Page]ExoUniversity of Freiburg
FPHA2018CVPR[Code] [Page]EgoImperial College London
VLOG2018CVPR[Code] [Page]ExoUniversity of Michigan
EPIC-KITCHENS2018ECCV[Code] [Page]EgoUniversity of Bristol
EGTEA Gaze+2018ECCV[Code] [Page]EgoUniversity of Wisconsin-Madison
YouCook22018AAAI[Code] [Page]ExoUniversity of Michigan
Something-Something2017ICCV[Code] [Page]ExoTwentyBN
Charades2016ECCV[Code] [Page]ExoCarnegie Mellon University
ActivityNet2015CVPR[Code] [Page]ExoUniversidad del Norte
EgoHands2015ICCV[Code] [Page]EgoIndiana University
Breakfast2014CVPR[Code] [Page]ExoFraunhofer FKIE

Human Video Generation

This section summarizes works that use video generation techniques to synthesize human videos for robot learning.

2026

  • ๐ŸŒŸ๐Ÿค– [ICLR] Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations [Code] [Page]
  • ๐ŸŒŸ [CVPR] Dexterous World Models [Code] [Page]
  • ๐ŸŒŸ๐Ÿค– [ICRA] NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos [Code] [Page]

2025

  • ๐ŸŒŸ๐Ÿค– [CoRL] Dreamitate: Real-World Visuomotor Policy Learning via Video Generation [Code] [Page]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.12] Large Video Planner Enables Generalizable Robot Control [Code] [Page]
  • ๐ŸŒŸ๐Ÿค– [arXiv 2025.12] Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow [Code] [Page]

2024

  • ๐Ÿค– [arXiv 2024.09] Gen2Act: Human Video Generation in Novel Scenarios Enables Generalizable Robot Manipulation [Code] [Page]

2023

  • ๐ŸŒŸ [NeurIPS] Learning Universal Policies via Text-Guided Video Generation [Code] [Page]

2020

  • ๐Ÿค– [RA-L] Learning One-Shot Imitation from Humans Without Humans [Code] [Page]

HOI Analysis Techniques

This section gathers HOI analysis techniques, including hand/object detection and reconstruction, pose estimation, and point tracking tools widely used in LfHV research.

2026

2025

  • ๐ŸŒŸ [CVPR] WiLoR: End-to-End 3D Hand Localization and Reconstruction In-the-Wild [Code] [Page]
  • ๐ŸŒŸ [CVPR] HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos [Code] [Page]
  • ๐ŸŒŸ [CVPR] Any6D: Model-Free 6D Pose Estimation of Novel Objects [Code] [Page]
  • ๐ŸŒŸ [CVPR] Structured 3D Latents for Scalable and Versatile 3D Generation [Code] [Page]
  • ๐ŸŒŸ [ICCV] CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos [Code] [Page]
  • ๐ŸŒŸ [ICCV] SpatialTrackerV2: Advancing 3D Point Tracking with Explicit Camera Motion [Code] [Page]
  • [Meshy AI] Meshy AI: The #1 AI 3D Model Generator for Creators [Page]

2024

  • ๐ŸŒŸ [CVPR] Reconstructing Hands in 3D with Transformers [Code] [Page]
  • ๐ŸŒŸ [CVPR] FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects [Code] [Page]
  • ๐ŸŒŸ [CVPR] SpatialTracker: Tracking Any 2D Pixels in 3D Space [Code] [Page]
  • ๐ŸŒŸ [ECCV] CoTracker: It Is Better to Track Together [Code] [Page]
  • ๐ŸŒŸ [ECCV] Local All-Pair Correspondence for Point Tracking [Code] [Page]
  • ๐ŸŒŸ [ACCV] Bootstap: Bootstrapped Training for Tracking-Any-Point [Code] [Page]
  • ๐ŸŒŸ [arXiv 2024.04] InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-View Large Reconstruction Models [Code] [Page]

2023

  • ๐ŸŒŸ [NeurIPS] Towards a Richer 2D Understanding of Hands at Scale [Code] [Page]
  • ๐ŸŒŸ [CVPR] BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects [Code] [Page]
  • ๐ŸŒŸ [ICCV] TAPIR: Tracking Any Point with Per-Frame Initialization and Temporal Refinement [Code] [Page]

2022

  • ๐ŸŒŸ [CVPR] Human Hands as Probes for Interactive Object Understanding [Code] [Page]
  • ๐ŸŒŸ [CoRL] MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare [Code] Page]
  • ๐ŸŒŸ [ICCVW] FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration [Code] [Page]

2020

  • ๐ŸŒŸ [CVPR] Understanding Human Hands in Contact at Internet Scale [Code] [Page]
  • ๐ŸŒŸ [CVPRW] MediaPipe Hands: On-Device Real-Time Hand Tracking [Code] [Page]

Other vision foundation models like Grounding DINO, YOLO series, SAM series are also widely used in the LfHV literature.

๐Ÿ“„ LICENSE

This project is licensed under the MIT License - see the LICENSE file for details.

๐Ÿ“ฌ Contact

If you have any questions or suggestions, please feel free to contact Junyi Ma.