Training data generation from Matterport3D dataset.

We augment Matterport3D with plane segmentation annotations on the original mesh. This enables consistent plane instance ID and plane parameters through rendering, and automatically establishes plane correspondences across different images. We fit planes using RANSAC, following PlaneNet, on mesh vertices within the same object instance (using the annotation provided by the original Matterport3D dataset). We then render the per-pixel plane instance ID, along with the RGB-D images using AI Habitat habitat-sim. Since the original instance segmentation mesh has “ghost” objects and holes due to artifacts, we further filter out bad plane annotations by comparing the depth information and the plane parameters.

Fit planes on Matterport3d meshes

You need to download Matterport3D dataset and Habitat-MP3D. We heavily used code from PlaneNet to fit planes.

# Fit plane
bash fit_plane_ply.bash

# Generate plane segment mesh
bash convert_mp3d_ply.bash

Render gt plane segmentation

Note: you need to install habitat-sim.

bash render_plane.bash