custom_object.md

Reconstructing custom objects

In the following, we will reconstruct a kettle object.

1. Download the images for the kettle object from here. kttle.zip contains only the images. kettle_processed.zip contains all the processed intermediate results.
2. Organize files like

   NeRO
   |-- data
       |-- custom
           |-- kettle
               |-- images
   

3. Run COLMAP to recover camera poses.

   # run colmap on the kettle and all images are captured by the same camera.
   python run_colmap.py --project_dir data/custom/kettle --colmap <path-to-your-colmap-exe> --same_camera
   

4. We need to specify the region we want to reconstruct. We achieve that by cropping a point cloud from the point cloud reconstructed by COLMAP. Note the point cloud is only used to compute the bounding sphere, but not for the reconstruction. As an example, we use the CloudCompare as the tool. The reconstructed point cloud locates at data/custom/kettle/colmap/point.ply. Export the cropped point cloud as object_point_cloud.ply. Place the object_point_cloud.ply at data/custom/kettle/object_point_cloud.ply. We provide a video here. LMB means Left-Mouse-Button, RMB means Right-Mouse-Button and MMB means Middle-Mouse-Button
5. Manually specify the forward (x+) direction and the up (z+) direction for the object. For the z+ direction, we may crop a horizontal point cloud first: Then, we fit a plane on the cropped point cloud. And its normal is the Z+ or Z- direction. Then, we use the point picking tool to specify the X+ direction. Write a data/custom/kettle/meta_info.txt with the content: The first line is the Z+ direction and the second line is the X+ direction. Your results may be different from mine since the reconstructed point cloud of COLMAP may have a different coordinate system. We provide a video here. LMB means Left-Mouse-Button, RMB means Right-Mouse-Button and MMB means Middle-Mouse-Button You may also refer to this issue and this markdown.
6. Make sure that you have the following files, which are generated by above steps.

   Gen6D
   |-- data
       |-- custom
          |-- kettle
              |-- object_point_cloud.ply  # object point cloud
              |-- meta_info.txt           # meta information about z+/x+ directions
              |-- images                  # images
              |-- colmap                  # colmap project
   

7. Run the shape reconstruction by

   python run_training.py --cfg configs/custom/kettle_shape.yaml
   

8. Extract the reconstructed mesh by

   python extract_mesh.py --cfg configs/custom/kettle_shape.yaml
   

   The resulted mesh will be saved at data/meshes/kettle_shape-300000.ply.
9. Run the material estimation by

   python run_training.py --cfg configs/custom/kettle_material.yaml
   

10. Extract the material by

python extract_materials.py --cfg configs/custom/kettle_material.yaml

The extracted materials will be saved at data/materials/kettle_material-100000.

11. Relighting

python relight.py --blender <path-to-your-blender> \
               --name kettle-neon \
               --mesh data/meshes/kettle_shape-300000.ply \
               --material data/materials/kettle_material-100000 \
               --hdr data/hdr/neon_photostudio_4k.exr

You should see images in the data/relight/kettle-neon.