DDLoc localization: a sim-to-real learning method for absolute localization

This repo contains the Pytorch implementation of DDLoc, an adapation of ARC method for absolute coordinate regression.

Please make sure you have access to the CrossLoc Benchmark Raw Datasets and have set it up properly before proceeding.

Also check out other useful repos regarding the datasets:

• CrossLoc-Benchmark-Datasets: CrossLoc benchmark datasets setup tutorial.
• TOPO-DataGen: an open and scalable aerial synthetic data generation workflow.

The DDLoc localization algorithm is officially presented in the paper accepted to CVPR 2022
CrossLoc: Scalable Aerial Localization Assisted by Multimodal Synthetic Data
Qi Yan, Jianhao Zheng, Simon Reding, Shanci Li, Iordan Doytchinov
École Polytechnique Fédérale de Lausanne (EPFL)
Links: website | arXiv | code repos | datasets

Happy coding! :)

Contents

• Requirments
• Training Precedures
• Evaluations
• Pretrained Models

Requirements

1. Python 3.6 with Ubuntu 16.04
2. Pytorch 1.1.0
3. dsacstar (if you want to test the camera pose estimation from the scene coordinate prediction)

You also need other third-party libraries, such as numpy, pillow, torchvision, and tensorboardX (optional) to run the code.

We suggest to follow the procedure in CrossLoc repo to install dependecies.

Datasets

You have to download our provided urbanscape data and place them in the following structure to load the data. See Pretrained Models section for download links.

Dataset Structure

urban (real)
    | train
        | rgb
        | poses
        | init
        | calibration
    | test
        | rgb
        | poses
        | init
        | calibration
urban (synthetic)
    | train
        | rgb
        | poses
        | init
        | calibration      

You can download naturescape data for more experiments and follow the same structure.

Training Precedures

• 1 Train Initial Coordinate Regressor C (train_C.py)
• 2 Train Style Translator T (train_T.py)
• 3 Train Initial Attention Module A (train_A.py)
• 4 Train Inpainting Module I (train_U.py)
• 5 Jointly Train Coordinate Regressor C and Attention Module A (train_joint_A_C.py)
• 6 Finetune the Coordinate Regressor C with translated image (train_finetune_C.py)

We provide example scripts for training each step in this folder batch_size and eval_batch_size are flexible to change given your working environment.

1 Train Initial Coordinate Regressor C

Train an initial coordinate regressor C with real and synthetic data. The best model is picked by the one with minimum camera poses error. The checkpoints are saved in ./checkpoints/your_dir_name/train_initial_coord_regressor_C/.

2 Train Style Translator T

Train the style translator T with naive mixed data and finetune T by paired real and synthetic data. The best model is picked by visual inspection & training loss curves.

3 Train Initial Attention Module A

Train an initial attention module A from scratch with descending $\tau$ values.

4 Train Inpainting Module I

Train the inpainting module I with T (from step 2) and A (from step 3).

5 Jointly Train Coordinate Regressor C and Attention Module A

Further jointly train coordinate regressor C and attention module A together with C (from step 1), T (from step 2), A (from step 3) and I (from step 4). The A and C learned from this step is the good initialization before finetuning C with coordinate regression loss and reprojecetion loss. In step 5, we train for relatively less epochs.

6 Finetune the Coordinate Regressor C with translated image

Lastly, we finetune the coordinate regressor C with oordinate regression loss and reprojecetion loss using C (from step 5). The training translated image is generated by T (from step 2), A (from step 5) and I (from step 4). generate_translated.py can be used to generate translated images from real images. An example is given in Generate_translated.sh

Evaluations

Evaluate the final results, you can make use of eval.py with an example given in eval.sh If you want to evaluate with your own data, please place your own data under <real dataset>/test with the dataset structure described above.

Pretrained Models

Pretrained network weights, the testing set images and the full datasets could be found as follows:

• oneDrive
• Google Drive
• Dryad (Full CrossLoc Benchmark Datasets only)

Please also refer to the general code repos for other details on the CrossLoc benchmark datasets.

Sample Result Visualization

Acknowledgments

This code is developed based on ARC and Pytorch-CycleGAN.

Citation

If you find our code useful for your research, please cite the paper:

@article{yan2021crossloc,
  title={CrossLoc: Scalable Aerial Localization Assisted by Multimodal Synthetic Data},
  author={Yan, Qi and Zheng, Jianhao and Reding, Simon and Li, Shanci and Doytchinov, Iordan},
  journal={arXiv preprint arXiv:2112.09081},
  year={2021}
}

@misc{iordan2022crossloc, 
	title={CrossLoc Benchmark Datasets}, 
	author={Doytchinov, Iordan and Yan, Qi and Zheng, Jianhao and Reding, Simon and Li, Shanci}, 
	publisher={Dryad}, 
	doi={10.5061/DRYAD.MGQNK991C}, 
	url={http://datadryad.org/stash/dataset/doi:10.5061/dryad.mgqnk991c},
	year={2022}
}