Implementation of RI3D - state of the art few-shot gaussian splatting.

Makes a gaussian splat from very few images (down to 3). Existing solutions (COLMAP + postshot) look terrible with few cameras - holes, floaters, and they're picky about camera placement. COLMAP needs ~12 cameras minimum for a full body scan and still produces results that need manual cleanup.

RI3D cleans up the floaters and inpaints holes with fine tuned stable diffusion models automatically.

---

Since we are just fine tuning models, you can train it yourself on a consumer level gpu. Works on my RTX 2060 Super with 8GB VRAM. Only tested on Cuda GPUs, may work on others too tho.

Dataset

I used the mip-nerf 360 dataset for training.

wget http://storage.googleapis.com/gresearch/refraw360/360_v2.zip -O 360_v2.zip
./utils/extract_dataset.sh

Downloads & extracts only the images from each scene in 360_v2 into dataset/. Requires p7zip (7z command).

Install

Install Python dependencies:

Install Pytorch following the install instructions.

Then install DUSt3R by following its README or using utils/install_dust3r.sh. Once it's set up, move its source folders into this repo so the following paths exist:

• src/dust3r_visloc
• src/dust3r
• src/croco

Then install the rest of the dependencies:

pip install -r requirements.txt

Usage

Make a new folder with your scene name in dataset. Then place your three images there. Then run the following:

python src/run_pipeline.py --prep --scene dataset/your-scene 
python src/run_pipeline.py --train --scene dataset/your-scene
python src/run_pipeline.py --optimize --scene dataset/your-scene

Note: you can omit --scene and it will run steps across all scenes it finds (in a memory efficient way)

--train runs all three training stages (LOO data, repair model, inpainting model). If you want to run only some of them, use the granular flags — they compose:

python src/run_pipeline.py --prep --scene dataset/your-scene      # prep only
python src/run_pipeline.py --train_loo --scene dataset/your-scene      # LOO pairs only
python src/run_pipeline.py --train_repair --scene dataset/your-scene   # repair model only (needs LOO pairs)
python src/run_pipeline.py --train_inpaint --scene dataset/your-scene  # inpainting model only
python src/run_pipeline.py --train_loo --train_repair --scene dataset/your-scene  # any combination

This takes quite a while. with 3 input views and RTX 2060S, 30-40min per scene. This can surely be improved a lot and im working on that while also getting the quality up at the same time. So its eventually usable for gsplat video :D

Running on RunPod / vast.ai

with runpod or vast.ai you can rent a beefy gpu server if you're impatient (like me). The runpod/ folder has helper scripts for this, see runpod/README.md for the full workflow.

Running on Modal

You can also run the pipeline on Modal. Modal makes it very easy and doesn't require messing with like server instances and copying files around. Check out the docstring at the top of src/run_modal.py for how to run it there.

Performance

Here are the run-times for prep + train_loo + train_repair across different hardware, so you can estimate how long it'll take on your own system:

Hardware	Time
RTX 2060 Super	36 min
Modal L40S	14 min
RTX 5090	13 min
RTX Pro 6000	7.5 min