MemE: Distractor-Free Novel View Synthesis

Official implementation of Distractor-Free Novel View Synthesis via Exploiting Memorization Effect in Optimization, ECCV 2024.

Project page · Demo video · Paper · Supplementary · Models

MemE is an unsupervised, plug-and-play module for distractor-free novel view synthesis. It exploits the memorization effect during optimization to separate clean scene content from transient distractors, and can be combined with both NeRF and 3D Gaussian Splatting.

Highlights

• No clean targets, semantic masks, or manual annotations.
• Hierarchical residual measurement at pixel and patch levels.
• Adaptive purity-distractor mixture modeling.
• Implementations for MemE-NeRF and the method components used by MemE-3DGS.
• Evaluated on RobustNeRF, Kubric, and PhotoTourism scenes.

Installation

The code is based on MultiNeRF and uses JAX.

conda create -n meme python=3.9
conda activate meme
pip install -r requirements.txt
git clone https://github.com/rmbrualla/pycolmap.git internal/pycolmap

Install the appropriate GPU-enabled JAX build for your CUDA environment before training.

Data

Prepare a scene in the MultiNeRF/LLFF format:

scene/
├── images/
└── sparse/0/

Download third-party datasets from their original providers and follow their respective licenses. Dataset files are not redistributed in this repository.

Training

bash scripts/train_meme.sh /path/to/scene /path/to/checkpoints

The default configuration is configs/4cards_a_25_gmm_250000.gin. You can pass another configuration as the third argument.

Evaluation

bash scripts/eval_meme.sh /path/to/scene /path/to/checkpoints

Pretrained models

Nine 250k-step checkpoints are available in the pretrained-model release: five Kubric scenes and four RobustNeRF scenes. See MODEL_ZOO.md for scene-specific download links, selected run IDs, color-corrected PSNR/SSIM/LPIPS results, and SHA-256 checksums.

Each archive extracts to a checkpoint_250000/ directory. Use that directory as the checkpoint path when evaluating the corresponding scene:

tar -xzf meme-robustnerf-balloon-step250000.tar.gz
bash scripts/eval_meme.sh /path/to/balloon ./checkpoint_250000 configs/4cards_a_25_gmm_250000.gin

Citation

@inproceedings{wang2024meme,
  title     = {Distractor-Free Novel View Synthesis via Exploiting Memorization Effect in Optimization},
  author    = {Wang, Yukun and Li, Kunhong and Chen, Minglin and Wang, Longguang and Zhou, Shunbo and Xue, Kaiwen and Guo, Yulan},
  booktitle = {European Conference on Computer Vision},
  pages     = {477--493},
  year      = {2024}
}

License and acknowledgement

This repository is released under the Apache License 2.0 and is derived from the Google Research MultiNeRF codebase. See LICENSE and NOTICE for details.