GVGS: Gaussian Visibility-Aware Multi-View Geometry for Accurate Surface Reconstruction

Anonymous submission.

We propose GVGS, a Gaussian Visibility-Aware Multi-View Geometry for Accurate Surface Reconstruction. GVGS leverages the visibility information from the 3D Gaussian primitives to enhance the multi-view geometry constraints, and achieves state-of-the-art performance on both DTU and Tanks and Temples datasets.

Results

The Chamfer Distance↓ on the DTU dataset

	24	37	40	55	63	65	69	83	97	105	106	110	114	118	122	Mean
GVGS	0.32	0.53	0.33	0.33	0.79	0.52	0.47	1.05	0.63	0.58	0.37	0.53	0.30	0.35	0.32	0.49

The F1 Score↑ on the TnT dataset

Scene	F1 Score
Barn	0.58
Caterpillar	0.47
Courthouse	0.24
Ignatius	0.81
Meetingroom	0.39
Truck	0.68
Mean	0.53

Installation

conda create -n gvgs python=3.8
conda activate gvgs

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 #replace your cuda version
pip install -r requirements.txt
pip install submodules/diff-plane-rasterization
pip install submodules/simple-knn

Dataset Preprocess

Data Download

Please download the preprocessed DTU dataset from 2DGS, and download the ground truth point clouds from the DTU dataset. Download the Tanks and Temples dataset preprocessed by GOF, and the GT of Tanks and Temples dataset from official webiste.

The data folder should like this:

data
├── DTU
│   ├── dtu
│   │   ├── scan24
│   │   │   ├── images
│   │   │   ├── mask
│   │   │   ├── sparse
│   │   │   ├── cameras_sphere.npz
│   │   │   └── cameras.npz
│   │   └── ...
│   ├── dtu_eval
│   │   ├── Points
│   │   │   └── stl
│   │   └── ObsMask
├── TNT
│   ├── TrainingSet
│   │   ├── Ignatius
│   │   │   ├── images
│   │   │   ├── sparse
│   │   │   ├── train
│   │   ├── ...
│   └── GT
│       ├── Ignatius
│       │   ├── images
│       │   ├── masks
│       │   ├── normals
│       │   ├── sparse
│       │   ├── Ignatius_COLMAP_SfM.log
│       │   ├── Ignatius.json
│       │   ├── Ignatius.ply
│       │   ├── Ignatius_trans.txt
│       │   ├── Ignatius_mapping_reference.txt
│       │   ├── meta.json
│       │   ├── pinhole_dict.json
│       │   ├── transforms.json
│       │   ├── run-colmap-geometric.sh
│       │   └── run-colmap-photometric.sh
│       └── ...

Monocular Depth Generation

For each dataset, monocular depth is generated following the procedure described in Citygs-X.

conda activate citygx-x
mkdir -p TNT/TrainingSet/${tntid}/train/depths
ln -s TNT/TrainingSet/${tntid}/images TNT/TrainingSet/${tntid}/train/rgbs
python Depth-Anything-V2/run.py --encoder vitl --pred-only --grayscale --img-path TNT/TrainingSet/${tntid}/train/rgbs --outdir TNT/TrainingSet/${tntid}/train/depths
python utils/make_depth_scale.py --base_dir TNT/TrainingSet/${tntid} --depths_dir TNT/TrainingSet/${tntid}/train/depths

Training and Evaluation

DTU

python scripts/run_dtu_multi_gpu.py \
    --out_base_path ./output_dtu \
    --data_base_path data/DTU/dtu \
    --eval_path data/DTU/dtu_eval/Points/stl \
    --out_name test 

TNT

python scripts/run_tnt_multi_gpu.py \
    --out_base_path ./output_tnt \
    --data_base_path data/TNT/TrainingSet \
    --eval_path data/TNT/GT \
    --out_name test 

Acknowledgements

This project builds upon 3D Gaussian Splatting (3DGS) and PGSR.
The Gaussian influence implementation is adapted from efficientgaussian.
Data preprocessing for the DTU and Tanks and Temples datasets follows the pipelines of 2DGS and GOF.
Evaluation on DTU and Tanks and Temples is conducted using scripts adapted from DTUeval-python, TanksAndTemples, and QGS.
Monocular depth priors are generated based on CityGS-X and Depth Anything V2.

We sincerely thank all the authors for releasing their excellent work and codebases.