MatchNeRF

Explicit Correspondence Matching for
Generalizable Neural Radiance Fields

Yuedong Chen  ·  Haofei Xu  ·  Qianyi Wu  ·  Chuanxia Zheng
Tat-Jen Cham  ·  Jianfei Cai

TPAMI 2025

Paper | Project Page | Code | Model & Data 🤗

Table of Contents

• Setup Environment
• Download Datasets
  • DTU (for both training and testing)
  • Blender (for testing only)
  • Real Forward Facing (for testing only)
  • Tanks and Temples (for testing only)
• Testing
• Training
• Rendering Video
• Use Your Own Data
• Miscellaneous

Setup Environment

This project is developed and tested on a CUDA11 device. For other CUDA version, manually update the requirements.txt file to match the settings before preceding.

git clone --recursive https://github.com/donydchen/matchnerf.git
cd matchnerf
conda create --name matchnerf python=3.8
conda activate matchnerf
pip install -r requirements.txt

Troubleshooting:

pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121

conda install ffmpeg

pip install scikit_image==0.19.2

Download Datasets

DTU (for both training and testing)

• Download the preprocessed DTU training data dtu_training.rar and Depth_raw.zip from original MVSNet repo.

• Extract 'Cameras/' and 'Rectified/' from the above downloaded 'dtu_training.rar', and extract 'Depths' from the 'Depth_raw.zip'. Link all three folders to data/DTU, which should then have the following structure

data/DTU/
    |__ Cameras/
    |__ Depths/
    |__ Rectified/

Blender (for testing only)

• Download nerf_synthetic.zip and extract to data/nerf_synthetic.

Real Forward Facing (for testing only)

• Download nerf_llff_data.zip and extract to data/nerf_llff_data.

Tanks and Temples (for testing only)

• Download tnt_data.zip and extract to data/tnt_data.

Testing

MVSNeRF Setting (3 Nearest Views)

Download the pretrained model matchnerf_3v.pth and save to configs/pretrained_models/matchnerf_3v.pth, then run

python test.py --yaml=test --name=matchnerf_3v

If encounters CUDA out-of-memory, please reduce the ray sampling number, e.g., append --nerf.rand_rays_test==4096 to the command.

Performance should be exactly the same as below,

Training

Download the GMFlow pretrained weight (gmflow_sintel-0c07dcb3.pth) from the original GMFlow repo, and save it to configs/pretrained_models/gmflow_sintel-0c07dcb3.pth, then run

python train.py --yaml=train

Rendering Video

python test.py --yaml=test_video --name=matchnerf_3v_video

Results (without any per-scene fine-tuning) should be similar as below,

Use Your Own Data

• Download the model (matchnerf_3v_ibr.pth) pretrained with IBRNet data (follow 'GPNR Setting 1'), and save it to configs/pretrained_models/matchnerf_3v_ibr.pth.
• Following the instructions detailed in the LLFF repo, use img2poses.py to recover camera poses.
• Update the colmap data loader at datasets/colmap.py accordingly.

We provide the following 3 input views demo for your reference.

# lower resolution but fast
python test.py --yaml=demo_own
# full version
python test.py --yaml=test_video_own

The generated video will look like,

Demo: own data, printer

Miscellaneous

Citation

If you use this project for your research, please cite our paper.

@article{chen2025explicit,
  title={Explicit correspondence matching for generalizable neural radiance fields},
  author={Chen, Yuedong and Xu, Haofei and Wu, Qianyi and Zheng, Chuanxia and Cham, Tat-Jen and Cai, Jianfei},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2025},
  publisher={IEEE}
}

Acknowledgments

This implementation borrowed many code snippets from GMFlow, MVSNeRF, BARF, GIRAFFE and MVSGaussian. Many thanks for all the above mentioned projects.