This repo includes the training and evaluation code of GVHMR on BEDLAM2 dataset.

Project Page | Paper

BEDLAM2.0: Synthetic Humans and Cameras in Motion

BEDLAM2.0 is a large-scale synthetic video dataset of animated bodies in simulated clothing, designed to train and test algorithms on the task of 3D human pose and shape estimation. With more than 8 million images, it is a significant expansion of the popular BEDLAM dataset that increases pose and body shape variation, and adds shoes and strand-based hair. Most importantly, it introduces a wide range of realistic cameras and camera motions.

GVHMR: World-Grounded Human Motion Recovery via Gravity-View Coordinates

Project Page | Paper

World-Grounded Human Motion Recovery via Gravity-View Coordinates
Zehong Shen^*, Huaijin Pi^*, Yan Xia, Zhi Cen, Sida Peng^†, Zechen Hu, Hujun Bao, Ruizhen Hu, Xiaowei Zhou
SIGGRAPH Asia 2024

Setup

Please see installation for details.

Quick Start

Google Colab demo for GVHMR

HuggingFace demo for GVHMR

Demo

Demo entries are provided in tools/demo. Use -s to skip visual odometry if you know the camera is static, otherwise the camera will be estimated by DPVO. We also provide a script demo_folder.py to inference a entire folder.

python tools/demo/demo.py --video=docs/example_video/tennis.mp4 -s
python tools/demo/demo_folder.py -f inputs/demo/folder_in -d outputs/demo/folder_out -s

Reproduce

Test:

To reproduce the 3DPW, RICH, EMDB, BEDLAM2 results in a single run, use the following command:

python tools/train.py --cfg_file hmr4d/configs/gvhmr_b1b2.yaml ckpt_path=inputs/checkpoints/gvhmr/gvhmr_b1b2.ckpt task=test

Train:

To train the model using BEDLAM1 and BEDLAM2 datasets use the following command:

# BEDLAM1 only
python tools/train.py --cfg_file hmr4d/configs/gvhmr_b1.yaml

# BEDLAM2 only
python tools/train.py --cfg_file hmr4d/configs/gvhmr_b2.yaml

# BEDLAM1 + BEDLAM2
python tools/train.py --cfg_file hmr4d/configs/gvhmr_b1b2.yaml

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{shen2024gvhmr,
  title={World-Grounded Human Motion Recovery via Gravity-View Coordinates},
  author={Shen, Zehong and Pi, Huaijin and Xia, Yan and Cen, Zhi and Peng, Sida and Hu, Zechen and Bao, Hujun and Hu, Ruizhen and Zhou, Xiaowei},
  booktitle={SIGGRAPH Asia Conference Proceedings},
  year={2024}
}

Acknowledgement

We thank the authors of WHAM, 4D-Humans, and ViTPose-Pytorch for their great works, without which our project/code would not be possible.