Datasets

For view synthesis experiments with Gaussian splatting, we mainly use RealEstate10K and DL3DV datasets. We provide the data processing scripts to convert the original datasets to pytorch chunk files which can be directly loaded with this codebase.

Expected folder structure:

├── datasets
│   ├── re10k
│   ├── ├── train
│   ├── ├── ├── 000000.torch
│   ├── ├── ├── ...
│   ├── ├── ├── index.json
│   ├── ├── test
│   ├── ├── ├── 000000.torch
│   ├── ├── ├── ...
│   ├── ├── ├── index.json
│   ├── dl3dv
│   ├── ├── train
│   ├── ├── ├── 000000.torch
│   ├── ├── ├── ...
│   ├── ├── ├── index.json
│   ├── ├── test
│   ├── ├── ├── 000000.torch
│   ├── ├── ├── ...
│   ├── ├── ├── index.json

It's recommended to create a symbolic link from YOUR_DATASET_PATH to datasets using

ln -s YOUR_DATASET_PATH datasets

Or you can specify your dataset path with dataset.roots=[YOUR_DATASET_PATH]/re10k and dataset.roots=[YOUR_DATASET_PATH]/dl3dv in the config.

We also provide instructions to convert additional datasets to the desired format.

RealEstate10K

For experiments on RealEstate10K, we primarily follow pixelSplat and MVSplat to train and evaluate on the 256x256 resolution.

Please refer to pixelSplat repo for acquiring the processed 360p (360x640) dataset.

If you would like to train and evaluate on the high-resolution RealEstate10K dataset, you will need to download the 720p (720x1280) version. Please refer to here for the downloading script. Note that the script by default downloads the 360p videos, you will need to modify the 360p to 720p in this line of code to download the 720p videos.

After downloading the 720p dataset, you can use the scripts here to convert the dataset to the desired format in this codebase.

Considering the full 720p dataset is quite large and may take time to download and process, we provide a preprocessed subset in .torch chunks (download) containing two test scenes to quickly run inference with our model.

DL3DV

For experiments on DL3DV, we train and evaluate at both 256x448 and 512x960 resolutions.

For the test set, we use the DL3DV-Benchmark split, which contains 140 scenes for evaluation. You can first use the script src/scripts/convert_dl3dv_test.py to convert the test set, and then run src/scripts/generate_dl3dv_index.py to generate the index.json file for the test set.

For the training set, we use the DL3DV-480p dataset (270x480 resolution), where the 140 scenes in the test set are excluded during processing the training set. After downloading the DL3DV-480p dataset, you can first use the script src/scripts/convert_dl3dv_train.py to convert the training set, and then run src/scripts/generate_dl3dv_index.py to generate the index.json file for the training set.

Please note that you will need to update the dataset paths in the aforementioned processing scripts.

If you would like to train and evaluate on the high-resolution DL3DV dataset, you will need to download the DL3DV-960P version (540x960 resolution). Simply follow the same procedure for data processing, but update the images_8 folder to images_4.

Please follow the DL3DV license if you use this dataset in your project and kindly reference the DL3DV paper.

Considering the full 480p and 960p datasets are quite large and may take time to download and process, we provide our preprocessed dl3dv-480p-chunks and dl3dv-960p-chunks. Please note that the released datasets are intended solely for research purposes. We disclaim any responsibility for the misuse, inappropriate use, or unethical application of the dataset by individuals or entities who download or access it. We kindly ask users to adhere to the DL3DV license.

ACID

We also evaluate our generalization on the ACID dataset. Note that we do not use the training set; you only need to download the test set (provided by pixelSplat repo) for evaluation.

Additional Datasets

If you would like to train and/or evaluate on additional datasets, just modify the data processing scripts to convert the dataset format. Kindly note the camera conventions used in this codebase.