GFPose: Learning 3D Human Pose Prior with Gradient Fields

This repo contains the official implementation for the CVPR2023 paper: GFPose: Learning 3D Human Pose Prior with Gradient Fields. With GFPose, you can learn 3D pose prior and perform 5 downstream tasks in one model.

by Hai Ci, Mingdong Wu, Wentao Zhu, Xiaoxuan Ma, Hao Dong, Fangwei Zhong and Yizhou Wang.

Project Site (Demo) | Arxiv

Main requirements

• python >= 3.9
• pytorch
• 3DPoseViewer (for visualization)

pip install -r requirements.txt

File structure

Download 3d keypoints here. For corresponding h36m images (optional), please register on the official website. Download the pretrained U3D model here. Organize data and checkpoints in the following structure.

${POSE_ROOT}
|-- configs
|-- lib
|-- run
|-- checkpoint
    |-- u3d
        |-- best_model.pth
|-- data
    |-- h36m
        |-- h36m_train.pkl
        |-- h36m_test.pkl
        |-- h36m_sh_dt_ft.pkl

Train the unified model (U3D) form scratch

CUDA_VISIBLE_DEVICES=0 python -m run.train_fc_adv_3d --config configs/subvp/h36m_ncsnpp_deep_continuous.py --name u3d

Evaluate on five different tasks

Task1 - 3D Pose Estimation

Sample 200 hypotheses with SDE.

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m run.eval_fc_mp_adv_3d --config configs/subvp/h36m_ncsnpp_deep_continuous.py --ckpt-dir checkpoint/u3d --best --sample 640 --gpus 4 --hypo 200 --save results est

Sample 1 hypothesis with probability flow ODE.

CUDA_VISIBLE_DEVICES=0 python -m run.eval_fc_mp_adv_3d --config configs/subvp/h36m_ncsnpp_deep_continuous.py --ckpt-dir checkpoint/u3d --best --sample 640 --gpus 1 --hypo 1 --pflow --save trajs est

Visualize

python -m run.visualize --num-human 1 --num-hypo 80 --vis-gt

Task2 - Pose Completion (Incomplete 2D -> 3D)

Evaluate with 200 hypotheses and 2 randomly selected missing joints.

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m run.eval_fc_mp_adv_3d --config configs/subvp/h36m_ncsnpp_deep_continuous.py --ckpt-dir checkpoint/u3d --best --sample 640 --gpus 4 --hypo 200 --save results comp2d --randj 2

Manually specified missing joints (e.g. right leg).

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m run.eval_fc_mp_adv_3d --config configs/subvp/h36m_ncsnpp_deep_continuous.py --ckpt-dir checkpoint/u3d --best --sample 640 --gpus 4 --hypo 200 --save results comp2d --jlist 1,2,3

Visualize

python -m run.visualize --num-human 1 --num-hypo 40 --vis-gt

Task3 - Pose Completion (Incomplete 3D -> 3D)

Evaluate with 1 hypothesis and 3 randomly selected missing joints.

CUDA_VISIBLE_DEVICES=0 python -m run.eval_fc_mp_adv_3d --config configs/subvp/h36m_ncsnpp_deep_continuous.py --ckpt-dir checkpoint/u3d --best --sample 640 --gpus 1 --hypo 1 --pflow --save trajs comp3d --randj 3

Manually specified missing joints (e.g. two legs).

CUDA_VISIBLE_DEVICES=0 python -m run.eval_fc_mp_adv_3d --config configs/subvp/h36m_ncsnpp_deep_continuous.py --ckpt-dir checkpoint/u3d --best --sample 640 --gpus 1 --hypo 1 --pflow --save trajs comp3d --jlist 1,2,3,4,5,6

Visualize

python -m run.visualize --num-human 1 --num-hypo 1 --vis-gt

Task4 - 3D Pose Generation

CUDA_VISIBLE_DEVICES=0 python -m run.eval_fc_mp_adv_3d --config configs/subvp/h36m_ncsnpp_deep_continuous.py --ckpt-dir checkpoint/u3d --best --sample 640 --gpus 1 --hypo 1 --save trajs gen

Visualize

python -m run.visualize --num-human 40 --num-hypo 1 --num-perline 5

Task5 - 3D Pose Denoising

Denoise mocap data.

CUDA_VISIBLE_DEVICES=0 python -m run.denoise_fc_adv_3d --config configs/subvp/h36m_ncsnpp_deep_continuous.py --ckpt-dir checkpoint/u3d --best --save den --noise-type gaussian --std 5 --t 0.05

Visualize

python -m run.visualize --num-human 1 --num-hypo 1 --vis-gt

GUI

This repo includes a modified version of 3DPoseViewer to visualize the multi-hypotheses and diffusion process. Please find the dependencies to run this viewer here.

python -m run.visualize
Args
  --num-human  # number of persons visualized in a batch
  --num-hypo  # number of sampled hypotheses to visualize
  --vis-gt  # whether to visualize humans
  --num-perline  # number of humans to plot in a line

Citation

If you find this code useful in your project, please consider citing:

@article{ci2022gfpose,
  title={GFPose: Learning 3D Human Pose Prior with Gradient Fields},
  author={Ci, Hai and Wu, Mingdong and Zhu, Wentao and Ma, Xiaoxuan and Dong, Hao and Zhong, Fangwei and Wang, Yizhou},
  journal={arXiv preprint arXiv:2212.08641},
  year={2022}
}

Acknowledgement

This repo is built on the excellent work score_sde by Yang Song.