Installation

:rocket: Environment

Get started

We test the installation with

• GPU NVIDIA GeForce RTX 3090
• Ubuntu 20.04 LTS
• CUDA == 11.3, V11.3.58

$ conda env create -f environment.yaml
$ conda activate cpf

Install dependencies

$ pip install -r requirements.txt
$ pip install -r requirements@git.txt
$ pip install pytorch3d --no-index --no-cache-dir --find-links https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py310_cu113_pyt1110/download.html

Install thirdparty

dex-ycb-toolkit

$ cd thirdparty

# create a __init__.py in dex_ycb_toolkit
$ touch ./dex-ycb-toolkit/dex_ycb_toolkit/__init__.py
$ pip install ./dex-ycb-toolkit

# check install success:
$ python -c "from dex_ycb_toolkit.dex_ycb import DexYCBDataset"

libmesh

$ cd thirdparty/libmesh

$ python setup.py build_ext --inplace

# check install success: (in the project root directory)
$ python -c "import thirdparty.libmesh"

 

:luggage: Assets

MANO hand model

Get the MANO hand model mano_v1_2.zip from the MANO website.

1. click Download on the top menu, this requires register & login.
2. on the Download page, navigate to Models & Code section, and click Models & Code, the mano_v1_2.zip will be downloaded automatically.
3. Unzip mano_v1_2.zip and copy it into an assets folder.

handobjectconsist assets

Download the contents of the handobjectconsist/assets, unzip, and put them in the assets folder.

the assets folder should have the following structure:

assets
├── anchor
│   ├── anchor_mapping_path.pkl
│   ├── anchor_weight.txt
│   ├── face_vertex_idx.txt
│   └── merged_vertex_assignment.txt
├── closed_hand
│   └── hand_mesh_close.obj
├── fhbhands_fits
│   ├── Subject_1
│   ├── ...
│   └── Subject_6
├── hand_palm_full.txt
├── mano
│   ├── fhb_skel_centeridx0.pkl
│   └── fhb_skel_centeridx9.pkl
├── mano_v1_2
│   ├── __init__.py
│   ├── LICENSE.txt
│   ├── models
│   └── webuser

 

:floppy_disk: Data Preparation

First-Person Hand Action Benchmark (fphab)

1. Download the fphab following the official instructions, and link it to data/fhbhands.

2. Resize the original full-res images based on the handobjectconsist/reduce_fphab.py.

3. Download our fhbhand supplimentary file fhbhands_supp.tar.gz, and link it to data/fhbhands_supp

The fphab-related data in the project should has the following structure:

data
├── fhbhands
│   ├── action_object_info.txt
│   ├── action_sequences_normalized
│   ├── change_log.txt
│   ├── data_split_action_recognition.txt
│   ├── file_system.jpg
│   ├── Hand_pose_annotation_v1
│   ├── Object_6D_pose_annotation_v1_1
│   ├── Object_models
│   ├── Subjects_info
│   └── Video_files_480
├── fhbhands_supp 
│   ├── Object_contact_region_annotation_v512
│   ├── Object_models
│   └── Object_models_binvox

HO3Dv2

1. Download the HO3D dataset (version 2) following the official instructions and link it to data/HO3D.
2. Download our YCB models supplimentary YCB_models_supp_v2.tar.gz and link it to data/YCB_models_supp_v2.
3. Download our synthetic data: HO3D_ycba.tar.gz and HO3D_syntht.tar.gz, link them to data/HO3D_ycba and data/HO3D_syntht.

The HO3D-related data in the project should has the following structure:

data
├── HO3D
│   ├── evaluation
│   ├── evaluation.txt
│   ├── train
│   └── train.txt
├── HO3D_supp_v2
│   ├── evaluation
│   └── train
├── HO3D_syntht
│   └── train
├── HO3D_ycba
│   ├── 003_cracker_box
│   ├── 004_sugar_box
│   ├── 006_mustard_bottle
│   ├── 010_potted_meat_can
│   ├── 011_banana
│   ├── 019_pitcher_base
│   ├── 021_bleach_cleanser
│   ├── 025_mug
│   ├── 035_power_drill
│   └── 037_scissors
├── YCB_models_supp_v2
│   ├── 002_master_chef_can
│   ├── 003_cracker_box
│   └── ...

DexYCB

We use the DexYCB to aid in training the contact recovery model.

1. Download DexYCB dataset from the official site, unzip and link the dataset to data/DexYCB.
2. Download our DexYCB supplimentary file DexYCB_supp.tar.gz and link it to data/DexYCB_supp.

The DexYCB-related data in the project should has the following structure:

data
├── DexYCB
│   ├── 20200709-subject-01
│   ├── ...
│   ├── 20201022-subject-10
│   ├── bop
│   ├── calibration
│   └── models
├── DexYCB_supp
│   ├── 20200709-subject-01
│   ├── ...
│   └── 20201022-subject-10

 

:snowflake: Saved Models

Download our model ckeckpoints at here and put the contents in the checkpoints folder.

The legacy folder contains the checkpoints for the hand-object pose estimation network (HoNet) and the contact recovery network (PiCR) in our ICCV 2021 conference paper.

The verified folder contains the checkpoints for the contact recovery network (PiCR) that is re-trained within this repository (the checkpoints used in:

• config/HONetPiCRPipeline_fphab.yml
• config/HOPose_PiCRPipeline_ho3dv2.yml