Visually-grounded Humanoid Agents

Hang Ye , Xiaoxuan Ma , Fan Lu , Wayne Wu , Kwan-Yee Lin , Yizhou Wang

Overview

Visually-grounded Humanoid Agents studies how digital humans can actively perceive, plan, and behave in realistic 3D scenes using only visual observations and specified goals. Our framework couples a World Layer for semantic 3D scene reconstruction and avatar instantiation with an Agent Layer for egocentric perception, embodied reasoning, and full-body action execution.

TODOs

We will open-source code, data, and benchmark. Stay tuned!

• Agent Layer inference code for avatar humanoid–scene interaction
• HSI benchmark: scenes, tasks, and evaluation scripts (SmallCity, XGRIDS, SAGE-3D)
• Generalization to more in-the-wild 3DGS assets (WorldLab, SuperSplat, Pointcosm, etc.)
• World Layer: occlusion-aware semantic scene reconstruction pipeline

Updates

• [2026-04-10] Paper released.
• [TBD] Code release.

Citation

If you find this work useful, please cite:

@article{ye2026vghuman,
  author  = {Ye, Hang and Ma, Xiaoxuan and Lu, Fan and Wu, Wayne and Lin, Kwan-Yee and Wang, Yizhou},
  title   = {Visually-grounded Humanoid Agents},
  journal = {arXiv preprint arXiv:2604.08509},
  year    = {2026},
}

Contact

• Hang Ye: yehang@pku.edu.cn