[NeurIPS 2024] TALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sight

February 27, 2026 · View on GitHub

This repository contains the official PyTorch implementation of the paper "TALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sight" paper (NeurIPS 2024) by Hyun-Kurl Jang* , Jihun Kim* and Hyeokjun Kweon*.

(* denotes equal contribution.)

[Paper]

News

  • TALoS is accepted at NeurIPS 2024 🎉🎉.
  • Official code and Paper will be released soon!
  • Official code is released on 2024.10.17
  • Paper is released on 2024.10.22

Introduction

Our main idea is simple yet effective: an observation made at one moment could serve as supervision for the SSC prediction at another moment. While traveling through an environment, an autonomous vehicle can continuously observe the overall scene structures, including objects that were previously occluded (or will be occluded later), which are concrete guidances for the adaptation of scene completion. Given the characteristics of the LiDAR sensor, an observation of a point at a specific spatial location at a specific moment confirms not only the occupation at that location itself but also the absence of obstacles along the line of sight from the sensor to that location. The proposed method, named Test-time Adaptation via Line of Sight (TALoS) , is designed to explicitly leverage these characteristics, obtaining self-supervision for geometric completion. Additionally, we extend the TALoS framework for semantic recognition, another key goal of SSC, by collecting the reliable regions only among the semantic segmentation results predicted at each moment. Further, to leverage valuable future information that is not accessible at the time of the current update, we devise a novel dual optimization scheme involving the model gradually updating across the temporal dimension.

Installation

  • PyTorch >= 1.10
  • pyyaml
  • Cython
  • tqdm
  • numba
  • Numpy-indexed
  • torch-scatter
  • spconv (tested with spconv==1.0 and cuda==11.3)

Data Preparation

SemanticKITTI

./
├── 
├── ...
├── model_load_dir
    ├──pretrained.pth
└── dataset/
    ├──sequences
        ├── 00/           
        │   ├── velodyne/	
        |   |	├── 000000.bin
        |   |	├── 000001.bin
        |   |	└── ...
        │   └── labels/ 
        |       ├── 000000.label
        |       ├── 000001.label
        |       └── ...
        │   └── voxels/ 
        |       ├── 000000.bin
        |       ├── 000000.label
        |       ├── 000000.invalid
        |       ├── 000000.occluded
        |       ├── 000001.bin
        |       ├── 000001.label
        |       ├── 000001.invalid
        |       ├── 000001.occluded
        |       └── ...
        ├── 08/ # for validation
        ├── 11/ # 11-21 for testing
        └── 21/
	    └── ...

Test-Time Adaptation

  1. Download SCPNet Pre-trained Models: Download the weights from this [link] and place them in the ./model_load_dir directory.
  2. (Optional) Baseline Results: Download the comparison results from this [link] and place them in ./experiments/baseline.
  3. Generate Predictions: Run the script to generate predictions on the dataset.

Validation set

python run_tta_val.py --do_adapt --do_cont --use_los --use_pgt

Test set

python run_tta_test.py --do_adapt --do_cont --use_los --use_pgt --sq_num={sequence number}

Evaluation

To evaluate test sequences in SemanticKITTI, you should submit the generated predictions to link. After generate predictions, prepare your submission in the designated format, as described in the competition page. Use the validation script from the semantic-kitti-api to ensure that the folder structure and number of label files in the zip file is correct.

Acknowledgements

We thanks for the open source project SCPNet.

Citation

If you use the codes, please cite the following publication:

@article{jang2024talos,
  title={TALoS: Enhancing semantic scene completion via test-time adaptation on the line of sight},
  author={Jang, Hyun-Kurl and Kim, Jihun and Kweon, Hyeokjun and Yoon, Kuk-Jin},
  journal={Advances in Neural Information Processing Systems},
  volume={37},
  pages={74211--74232},
  year={2024}
}