GS-EVT: Cross-Modal Event Camera Tracking based on Gaussian Splatting

Tao Liu    Runze Yuan    Yi'ang Ju    Xun Xu    Jiaqi Yang    Xiangting Meng    Xavier Lagorce    Laurent Kneip   

ShanghaiTech University

ArXiv | Video | Test Data

Over the past year and a half, Gaussian Splatting has garnered significant attention and is now recognized as a groundbreaking advancement in 3D Reconstruction. Consequently, the integration of 3DGS with event cameras to improve localization has become a key area of research.

To the best of our knowledge, we are proud to introduce the first Gaussian Splatting map-based event camera tracking framework, which leverages the differentiable and realistic novel view synthesis capability of 3DGS to enhance event camera tracking.

How to install

git clone https://github.com/ChillTerry/GS-EVT.git --recursive
cd GSEVT
conda create --name GSEVT python=3.7
conda activate GSEVT
pip install -r requirements.txt
cd submodules/diff-gaussian-rasterization/
pip install -e .

How to get the dataset

We have preprocessed the VECTOR dataset, so you can download here and and put it in ./data/VECTOR.

How to run a sequence

python main.py -c configs/VECTOR/desk_normal1_config.yaml

Structure of this project

GSEVT
├── configs                                 // Store config yamls for each sequence
├── data                                    // Store data of gs-splat maps and event data(in txt)
├── gaussian_splatting                      // original gaussian splatting modules
├── results                                 // tracking outputs
├── scripts                                 // shell scripts for batch run on server
│   ├── desk_batch_run.sh
│   ├── robot_batch_run.sh
│   └── sofa_batch_run.sh
├── submodules
│   └── diff-gaussian-rasterization         // customized rasterization module for this project
├── test                                    // test scripts
│   ├── test_event_camera.py
│   ├── test_event_rgb_overlay.py
│   ├── test_initial_pose.py
│   ├── test_intensity_change.py
│   └── test_single_frame_tracking.py
├── utils
│   ├── auxiliary.py
│   ├── event_camera
│   ├── pose.py                             // functions for calculating pose
│   ├── render_camera                       // defines Camera class and RenderFrame class
│   ├── tracker.py                          // code for tracking
│   └── visualizer.py                       // code for visualing tracking results
├── main.py                                 // run this file to start the tracking process
├── LICENSE
├── readme.txt
└── requirements.txt                        // packages needed for this project

Citation

@article{liu2024gs,
  title={GS-EVT: Cross-Modal Event Camera Tracking based on Gaussian Splatting},
  author={Liu, Tao and Yuan, Runze and Ju, Yi'ang and Xu, Xun and Yang, Jiaqi and Meng, Xiangting and Lagorce, Xavier and Kneip, Laurent},
  journal={arXiv preprint arXiv:2409.19228},
  year={2024}
}