Robust 3D Gaussian Splatting

Paul Ungermann, Armin Ettenhofer, Matthias Nießner, Barbara Rössle (Technical University of Munich)

---

Abstract: Neural radiance fields (NeRFs) are vulnerable to dynamic objects polluting the input data of an otherwise static scene that should not be part of the learned scene representation called distractors. In our project, we adapt 3D Gaussian Splatting, a novel method to generate NeRFs, to be robust against such distractors. We achieve this by implementing a robust loss function. We create segmentation masks of distractor to ignore them during training. Using this robust loss and our adaptations, we obtain significantly better quantitative and qualitative results on distractor-polluted scenes than the original implementation of 3D Gaussian Splatting and RobustNeRF.

Quantitative Results

Scene	Gaussian Splat.	RobustNeRF	+Neural	+Segment.	+Both
Statue	0.1443	0.1374	0.1376	0.1526	0.1214
Yoda	0.2001	0.2177	0.1746	0.1891	0.1684
And-bot	0.1638	0.1622	0.1308	0.1778	0.1314
Crab	0.1414	0.1919	0.1184	0.1765	0.1151
Mean	0.1624	0.1773	0.1404	0.1740	0.1341

LPIPS score for each scene, lower better. We can see that our method significantly performs better.

Qualitative Results

We conduct five different versions to evaluate our method:

• Gaussian Splat. (Baseline) The unmodified implementation of 3D Gaussian Splatting
• RobustNeRF: A version of RobustNeRF implemented for 3D Gaussian Splatting.
• RobustNeRF + Segmentation (+Segment.): Sequentially adds the segmentation overlap improvements after calculating the mask using RobustNeRF but doesn’t use logistic regression.
• RobustNeRF + Linear Layer (+Neural): Sequentially adds a trainable linear layer after calculating the mask using RobustNeRF but doesn’t use segmentation.
• RobustNeRF + Linear Layer + Segmentation (+Both): Our method.

Gaussian Splat.	RobustNeRF	+Neural	+Segment.	+Both

Technical Tutorial

How to run 3D Gaussian Splatting in general see here. In addition to these features, we provide a config file to set parameters for robust training. Use git clone https://github.com/ppaulee/Robust3DGaussians.git --recursive to clone the repository.

config.json:

key	value
save_mask_interval	interval (in number of interations) after a the mask should be saved (in /output//masks)
test_size	number of test images (assuming sorted images), if negative it uses every -n-th image for the test set
mask_start_epoch	number of epochs to start using the distractor masks
use_segmentation	flag to intersect the masks (true) or just using the raw masks (false)
seg_overlap	indicates the minimum intersection ratio to classify an object as a distractor
n_residuals	number of residuals from the last n iterations used to compute the mask
per_channel	indicates if the mask is computed using all three image channel (true)
use_neural	indicates if the neural approach will be used. If set to false you need to set per_channel to false as well!
lambda_reg	regularization parameter for the mask loss

Data Preprocessing

Before training the model you need to generate the object segmentation masks using SegmentAnything. You can use python3 segment.py --input <input path (use correct scale images!)> --output <path to dataset> --model-type vit_b --checkpoint <path>.ckpt to generate the object segmentation mask. Note that you need to install SegmentAnything and use the latest vit_b checkpoint. See here for a tutorial.