3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors

(NeurIPS 2024 $\textcolor{red}{Spotlight}$)

Paper | Project Page

Xi Liu, Chaoyi Zhou, Siyu Huang

Abstract

Abstract: Novel-view synthesis aims to generate novel views of a scene from multiple input images or videos, and recent advancements like 3D Gaussian splatting (3DGS) have achieved notable success in producing photorealistic renderings with efficient pipelines. However, generating high-quality novel views under challenging settings, such as sparse input views, remains difficult due to insufficient information in under-sampled areas, often resulting in noticeable artifacts. This paper presents 3DGS-Enhancer, a novel pipeline for enhancing the representation quality of 3DGS representations. We leverage 2D video diffusion priors to address the challenging 3D view consistency problem, reformulating it as achieving temporal consistency within a video generation process. 3DGS-Enhancer restores view-consistent latent features of rendered novel views and integrates them with the input views through a spatial-temporal decoder. The enhanced views are then used to fine-tune the initial 3DGS model, significantly improving its rendering performance. Extensive experiments on large-scale datasets of unbounded scenes.

Method

TL;DR: We propose a method that exploits view-consistent 2D generative priors, i.e., a video diffusion model, to enhance 3D Gaussian splatting rendering quality.

Code Released

1. 3dgs_dataset_generator: For paired dataset generation.

2. SVDFor3D: Train SVD for 3DGS-Enhancer with paired dataset. The pretrained weight is here

Not available yet

1. STD: Spatial-temporal decoder

2. Confidence-aware 3DGS

Reference

If you find our work helpful, please consider citing:

@inproceedings{liu20243dgs,
  title = {3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-Consistent 2D Diffusion Priors},
  author = {Liu, Xi and Zhou, Chaoyi and Huang, Siyu},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2024}
}