TAG: Text Prompt Augmentation for Zero-Shot Out-of-Distribution Detection

October 10, 2024 ยท View on GitHub

This is the offical repository for our TAG and the presentation can be found here.

Illustration

Arch_figure

Data Preparation

We consider the following (in-distribution) datasets: CIFAR-100, ImageNet-100, and ImageNet-1k.

Small-scale OOD datasets For small-scale ID (e.g. CIFAR-100), we use SVHN, Textures (dtd), Places365, LSUN and iSUN.

OOD datasets can be downloaded via the following links (source: ATOM and CIDER):

  • SVHN: download it and place it in the folder of datasets/small_OOD_dataset/svhn. Then run python utils/select_svhn_data.py to generate test subset.
  • Textures: download it and place it in the folder of datasets/small_OOD_dataset/dtd.
  • Places365: download it and place it in the folder of datasets/ood_datasets/places365/test_subset. We randomly sample 10,000 images from the original test dataset.
  • LSUN-C: download it and place it in the folder of datasets/small_OOD_dataset/LSUN.
  • iSUN: download it and place it in the folder of datasets/small_OOD_dataset/iSUN.

Large-scale OOD datasets For large-scale ID (e.g. ImageNet-100), we use the curated 4 OOD datasets from OpenImage-O, Texture, iNaturalist, and ImageNet-O. (source: ViM)

  • ImageNet. The ILSVRC 2012 dataset as In-distribution (ID) dataset.
  • ImageNet-100.
  • OpenImage-O. The OpenImage-O dataset is a subset of the OpenImage-V3 testing set. The filelist is here.
  • Texture. We rule out four classes that coincides with ImageNet. The filelist used in the paper is here.
  • iNaturalist. Follow the instructions in the link to prepare the iNaturalist OOD dataset.
  • ImageNet-O. Follow the guide to download the ImageNet-O OOD dataset.

Evaluation

  1. extract features 
    python eval/extract_features.py  # specify model name ['ViT-B/16', 'ViT-B/32','ViT-L/14', 'RN50', 'RN101']
  2. evaluation 
    python eval/eval_ood_cifar_100.py --model_name  ['ViT-B/16', 'ViT-B/32','ViT-L/14', 'RN50', 'RN101'] --shuffle --M [int_number_of_augmentations] --temp [temperature] --score_name ['MSP', 'Max-Logit','Energy','GEN']
python eval/eval_ood_imagenet_100.py --model_name  ['ViT-B/16', 'ViT-B/32','ViT-L/14', 'RN50', 'RN101'] --shuffle --M [int_number_of_augmentations] --temp [temperature] --score_name ['MSP', 'Max-Logit','Energy','GEN'] 
python eval/eval_ood_imagenet_1k.py --model_name  ['ViT-B/16', 'ViT-B/32','ViT-L/14', 'RN50', 'RN101'] --shuffle --M [int_number_of_augmentations] --temp [temperature] --score_name ['MSP', 'Max-Logit','Energy','GEN']
  3. texture embedding analysis 
     python eval/SVD_text_embeddings.py

Citation

@inproceedings{Liu2024TAG,
title = {TAG: Text Prompt Augmentation for Zero-Shot Out-of-Distribution Detection},
author = {Liu, Xixi and Christopher, Zach},
booktitle = {European Conference on Computer Vision},
year = {2024}
}

Acknowledgement

Our data preparartion code is adapted from ViM and CIDER repos, thanks a lot for their great work!