Contrastive Gaussian Clustering

This repository contains the implementation associated with the paper "Contrastive Gaussian Clustering: Weakly Supervised 3D Scene Segmentation", click in the next link for more details.

Contrastive Gaussian Clustering: Weakly Supervised 3D Scene Segmentation
Myrna C. Silva, Mahtab Dahaghin, Matteo Toso, Alessio Del Bue
Istituto Italiano di Tecnologia - IIT

Standard Installation

Clone the repository locally

git clone https://github.com/MyrnaCCS/contrastive-gaussian-clustering.git
cd contrastive-gaussian-clustering

We provide a conda environment setup file including all the dependencies. Create the conda environment contrastive-gaussian by running:

conda create -n contrastive-gaussian python=3.11 -y
conda activate contrastive-gaussian

pip install pytorch==2.3.0  torchvision==0.18.0  torchaudio==2.3.0
pip install plyfile==1.0.3
pip install tqdm scipy wandb opencv-python scikit-learn lpips

pip install submodules/diff-gaussian-rasterization
pip install submodules/simple-knn

Optimizing a CGC

To optimize a Contrastive Gaussian Clustering model:

python train.py -s <path to COLMAP or NeRF Synthetic dataset> -m <path where the trained model should be stored>

Rendering a CGC

To render a Contrastive Gaussian Clustering model:

python train.py -m <path to trained model>

ToDo List

• release the code to preprocess multi-view images
• release the code for open-vocabulary segmentation
• release the evaluation code
• release the preprocessed dataset and the pretrained model
• release the code for 3D object segmentation

Citation

@Article{contrastive_gaussians_2024,
      author       = {Myrna C. Silva and Mahtab Dahaghin and Matteo Toso and Alessio Del Bue},
      title        = {Contrastive Gaussian Clustering: Weakly Supervised 3D Scene Segmentation},
      journal      = {https://arxiv.org/abs/2404.12784},
      year         = {2024}
}

Thanks

This code is based on 3D Gaussian Splatting and Gaussian Grouping codebases. We thank the authors for releasing their code.