WildHOI: Monocular Human-Object Reconstruction in the Wild

Implementation for the paper: Monocular Human-Object Reconstruction in the Wild.

Set up Environments

1. create env. for pytorch and pytorch3d.

conda create -n wildhoi python=3.9
conda activate wildhoi
conda install pytorch==1.13.0 torchvision==0.14.0 torchaudio==0.13.0 pytorch-cuda=11.7 -c pytorch -c nvidia
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install -c bottler nvidiacub
conda install pytorch3d -c pytorch3d

2. install necessary packages.

pip install scipy opencv-python trimesh smplx scikit-image scikit-learn

3. compile neural renderer.

cd CODE_DIR
mkdir externals
cd ./externals
git clone https://github.com/JiangWenPL/multiperson.git

add the following lines

#ifndef AT_CHECK 
#define AT_CHECK TORCH_CHECK 
#endif 

at the beginning of files multiperson/neural_renderer/neural_renderer/cuda/*.cpp. Then compile it,

export CUDA_HOME=/public/software/CUDA/cuda-11.3/ (modify this path with your custom setting)
cd multiperson/neural_renderer && python setup.py install

Demo

1. download extra data and put them in the folder CODE_DIR/data.

2. Prepare SMPL-X by following the instructions and put them into the folder CODE_DIR/data.

3. download pre-trained model and put it in the folder CODE_DIR/outputs/cflow_pseudo_kps2d.

4. run

   python ./demo.py
   

   the results are saved to the path CODE_DIR/outputs/demo.

Train

1. download the WildHOI dataset from here.
2. generate KPS groups list. (or download them from here, put them in the folder CODE_DIR/outputs).

python KNN_grouping_wildhoi.py --root_dir DATASET_ROOT --object barbell --n_steps 100

3. Train the normalizing flow. (the pre-trained models are here)

python train_kps_flow_wildhoi.py --root_dir DATASET_ROOT --object barbell --batch_size 64 --dropout_probability 0.5

4. [Optional] Generate the mean occlusion map. (the pre-generated occlusion maps are here)

python get_occlusion_map --root_dir DATASET_ROOT --object barbell

Inference

Optimize the test images:

python optimize_with_kps_flow_imagewise.py --root_dir DATASET_ROOT --object barbell
python optimize_with_kps_flow_contact_imagewise.py --root_dir DATASET_ROOT --object barbell
python optimize_with_phosa.py --root_dir DATASET_ROOT --object barbell

Results can be visualized via:

python visualize_recon_results.py --root_dir DATASET_ROOT --object barbell --exp optim_with_kps_flow

Evaluation

Quantitative evaluation:

python evaluate_recon_results.py --exp optim_with_kps_flow --object barbell

Results:

Comparison on WildHOI-test Dataset

Method	Chamfer Distance		Object Pose Error
	SMPL(cm) ↓	Object(cm) ↓	Rot.(°) ↓	Transl.(cm) ↓
Isolation	3.52	1259.47	7.10	658.10
PHOSA	4.77	51.22	11.94	33.88
KPS Flow (w/o contact)	4.44	19.54	10.05	14.35
KPS Flow (with contact)	4.49	17.76	9.76	13.28

Objectwise Comparison on WildHOI-test Dataset

Metrics	Method	Barbell	Baseball	Basketball	Bicycle	Cello	Skateboard	Tennis	Violin	Average
SMPL(cm)↓	PHOSA	4.50	3.36	3.62	7.24	7.58	5.35	3.91	4.58	4.77
	Ours	5.08	4.09	4.26	4.75	6.18	4.19	4.47	4.12	4.49
Object(cm)↓	PHOSA	44.62	36.63	74.19	103.30	35.70	52.25	38.82	57.46	51.22
	Ours	8.17	30.31	22.98	12.96	8.08	12.45	21.16	6.62	17.76
Rot.(°) ↓	PHOSA	2.56	26.49	-	3.59	4.95	3.99	20.37	3.25	11.94
	Ours	3.47	24.30	-	3.17	5.28	4.18	10.97	3.01	9.76
Transl.(cm) ↓	PHOSA	30.53	22.18	44.29	69.83	26.40	35.70	26.04	37.84	33.88
	Ours	8.68	18.20	16.95	11.18	6.41	12.64	14.97	7.11	13.28

Qualitative evaluation:

python human_evaluation.py --root_dir DATASET_ROOT/OBJECT_NAME