README.md

February 28, 2024 · View on GitHub

HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning

Xiaozheng Zheng^1,2† Chao Wen^2† Zhou Xue² Pengfei Ren^1,2 Jingyu Wang^1*

¹Beijing University of Posts and Telecommunications ²PICO IDL ByteDance

^†Equal contribution ^*Corresponding author

:star_struck: Accepted to ICCV 2023

HaMuCo is a multi-view self-supervised 3D hand pose estimation method that only requires 2D pseudo labels for training.

[Project Page] • [arXiv]

:black_square_button: TODO

FreiHAND evaluation code
Multi-view inference code

:mega: Updates

[07/2023] HaMuCo is accepted to ICCV 2023 :partying_face:!

[01/2023] Training and evaluation codes on HanCo are released.

:file_folder: Data Preparation

1. Download the HanCo dataset from the official website.

2. We provide the 2D pseudo labels generated from OpenPose in `./data/HanCo/HaMuCo_*.zip`.

3. Unzip files and organize the data as follows:

${ROOT}  
|-- data  
|   |-- HanCo
|   |   |-- calib
|   |   |-- rgb 
|   |   |-- rgb_2d_keypoints
|   |   |-- rgb_merged
|   |   |-- xyz

:desktop_computer: Installation

Requirements

Python=3.7
PyTorch=1.9.1+cu111
torchgeometry (need some slight changes following here.)

Setup with Conda

conda create -n hamuco python=3.7
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
cd HaMuCo
pip install -r ./requirements.txt

:running_woman: Training

1. Run `./train.py` to train and evaluate on the HanCo dataset.

:love_you_gesture: Citation

If you find our work useful for your research, please consider citing the paper:

@inproceedings{
  zheng2023hamuco,
  title={HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning},
  author={Zheng, Xiaozheng and Wen, Chao and Xue, Zhou and Ren, Pengfei and Wang, Jingyu},
  booktitle={Proceedings of the IEEE/CVF international conference on computer vision},
  year={2023}
}

:newspaper_roll: License

Distributed under the MIT License. See LICENSE for more information.

:raised_hands: Acknowledgements

The pytorch implementation of MANO is based on manopth. We thank the authors for their great job!