AUCSeg: AUC-oriented Pixel-level Long-tail Semantic Segmentation (NeurIPS 2024)

Author: Boyu Han, Qianqian Xu*, Zhiyong Yang, Shilong Bao, Peisong Wen, Yangbangyan Jiang, Qingming Huang*

✨ Updates

[2024-10-3] 🔥 Released our AUCSeg code. We warmly welcome everyone to use it and give feedback or suggestions!

[2024-9-26] Our paper has been accepted to NeurIPS 2024.

🔧 Installation

• Clone the repository

git clone https://github.com/boyuh/AUCSeg.git

• Install the required libraries according to the guidelines in MMSegmentation - get_started.
• Download ADE20K, cityscapes, and coco_stuff164k according to the guidelines in MMSegmentation - dataset_prepare.

• (Optional) If you want to use SegNeXt as the backbone, download the mscan-l pretrained model from Our - Google Drive or SegNeXt - TsingHua Cloud.

🖥️ Training

AUCSeg is a plug-and-play component that can be integrated into almost any existing backbone with just two simple modifications:

1. Add the AUCLoss module (options include SquareAUCLoss, HingeAUCLoss, and ExpAUCLoss) to the loss_decode in the model.
2. Add the TMemoryBank module to the train_pipeline.

Below, we use SegNeXt-large as an example backbone to demonstrate how to incorporate AUCSeg for training:

For the ADE20K dataset, run this command:

python tools/train.py aucseg_configs/aucseg.segnext.large.512x512.ade.160k.py --work-dir results/aucseg.segnext.large.512x512.ade.160k --seed 0 --gpu-id 0

For the Cityscapes dataset, run this command:

python tools/train.py aucseg_configs/aucseg.segnext.large.1024x1024.city.160k.py --work-dir results/aucseg.segnext.large.1024x1024.city.160k --seed 0 --gpu-id 0

For the COCO-Stuff 164K dataset, run this command:

python tools/train.py aucseg_configs/aucseg.segnext.large.512x512.coco_stuff164k.80k.py --work-dir results/aucseg.segnext.large.512x512.coco_stuff164k.80k --seed 0 --gpu-id 0

⭐ Released Weights

We provide the Config and Checkpoint for three datasets.

Dataset	Backbone	Config	Checkpoint
ADE20K	SegNeXt-large	config	Google Drive
Cityscapes	SegNeXt-large	config	Google Drive
COCO-Stuff 164K	SegNeXt-large	config	Google Drive

📏 Evaluation

To evaluate the model, run this command:

./tools/dist_test.sh /path/to/config /path/to/checkpoint_file 1 --eval mIoU

✒️ Citation

If you find our work inspiring or use our codebase in your research, please cite our work.

@inproceedings{han2024aucseg,
    title={AUCSeg: AUC-oriented Pixel-level Long-tail Semantic Segmentation}, 
    author={Boyu Han and Qianqian Xu and Zhiyong Yang and Shilong Bao and Peisong Wen and Yangbangyan Jiang and Qingming Huang},
    booktitle={Advances in Neural Information Processing Systems},
    pages={126863--126907},
    year={2024}
}

💬 Contact

If you find any issues or plan to contribute back bug-fixes, please contact us by Boyu Han (Email: hanboyu23z@ict.ac.cn).

📚 Acknowledgement

Our codes are based on MMSegmentaion v0.24.1, SegNeXt and XCurve v1.1.0.