Using pointersect

Rendering point cloud from command line

We provide an example to render a point cloud stored as an obj file.

# at terminal
pointersect --input_point_cloud your_pcd.ply --output_dir out 

For more information, see the CLI doc.

Rendering point cloud within python

We provide functions to render with pointersect.

For example, the following function renders a point cloud:

def render_point_cloud_camera_using_pointersect(
        model_filename: str,  # checkpoint of the pointersect model
        k: int,  # number of neighboring points to use per ray
        point_cloud: PointCloud,
        output_cameras: Camera,  # camera to produce the query rays 
        max_ray_chunk_size: int = int(4e4),  # used to chunk the query rays to avoid out of memory
        model: SimplePointersect = None,  # preloaded pointersect model
) -> PointersectRecord:

For more information, see the api doc.

Train a new pointersect model on your dataset

We train the pointersect model on 48 meshes on sketchfab. Please see the mesh credit in the paper. We do not provide the download and pre-processing script. To train on these meshes, please download the meshes yourself. In addition, Open3D does not texturize some of these meshes properly. We recommend create new uv-mapped meshes using xatlas.

We provide a meta script to compile a config file and command to train a pointersect model.

# at the repo root
PYTHONPATH=. python pointersect/meta_script/submit_pbnr_v2.py

It creates a config file pointersect/meta_script/configs/pbnr.yaml to setup the training code. It also outputs a command to run the training script, for example:

# at the repo root
source /miniconda/etc/profile.d/conda.sh; conda activate /venv; CUDA_HOME="/venv/pkgs/cuda-toolkit" PYTHONPATH="." xvfb-run -a -s "-screen 0 800x600x24" torchrun --standalone --nnodes=1 --nproc_per_node=8 pointersect/script/train_v2.py --config_filename pointersect/meta_script/configs/pbnr.yaml

To train a new pointersect model on your datasets, you can customize the file pointersect/meta_script/configs/mesh_filenames.yaml to list the mesh filenames.

Environment

We use Python 3.7.9, PyTorch 1.10.1, and CUDA 11.3, but there is no specific requirements for the exact versions.
We do, however, require a CUDA environment that can compile cuda code (i.e., nvcc can be called).

As a reference, we provide the dependent packages extracted via pipreqs in requirements.txt.
As far as we know, there is no specific dependency on the package versions.

Note that you need to install PyTorch first before installing the pip-installable via

# at repo root
pip install .

or the dependencies via

# at repo root
pip install -r env/requirements.txt
pip install . 

Additionally, you may need to install Open3D in headless mode if you run the code on a server without a display.