GROOD: Gradient-Aware Out-of-Distribution Detection

This repository provides the official PyTorch implementation of:

GROOD: Gradient-Aware Out-of-Distribution Detection

Mostafa ElAraby, et al.

---

Abstract

Out-of-Distribution (OOD) detection is critical for deploying neural networks safely. GROOD introduces a gradient-aware framework that leverages gradients with respect to a synthetic OOD prototype as a discriminative signal. Unlike conventional methods that rely solely on feature distances or logits, GROOD projects samples into gradient space, where ID samples yield coherent, low-sensitivity patterns while OOD samples form outliers in magnitude and direction.

To operationalize this, GROOD:

• Constructs an OOD prototype from synthetic or auxiliary data.
• Computes gradients of the NCP softmax loss w.r.t. the OOD prototype.
• Uses nearest-neighbor scoring in gradient space to distinguish ID/OOD.

This approach achieves state-of-the-art performance across CIFAR-10, CIFAR-100, ImageNet-200, and ImageNet-1K benchmarks.

---

Highlights in OpenOOD v1.5

• CIFAR-10: Best near-OOD AUROC (91.16%).
• CIFAR-100: Strong far-OOD performance (84.44%).
• ImageNet-200: Competitive across both near/far OOD.
• ImageNet-1K: Outperforms prior methods with 78.9% near-OOD AUROC.

---

Method Overview

The GROOD methodology is illustrated in the following diagram:

• Input: An input sample.
• Feature Extractor: A pre-trained neural network (f_pen) extracts features.
• Feature Representation: The extracted features (h).
• Distance to Prototypes: Distances between the feature representation and the ID class prototypes (p_dog, p_cat) and the OOD prototype (p_OOD) are calculated.
• Gradients: Gradients of the softmax loss are computed with respect to the OOD prototype.
• OOD Score: The final OOD score is determined using nearest neighbor distance in the gradient space.

---

How to Run

The code provides a bash script (run.sh) to run GROOD on several datasets. You can use the run.sh script to run any of the experiments by specifying the i variable.

Prerequisites:

• Python 3.8
• CUDA 11.1
• OpenOOD v2
• Wandb (for experiment tracking)
• Required python packages (see the paper or environment.yml)

Datasets:

The script supports the following datasets:

• MNIST
• CIFAR-10
• CIFAR-100
• ImageNet (ResNet-50, ViT, Swin)
• ImageNet-200 (ResNet-18)

Running the Experiments:

1. Set up the environment:

   • Install the required dependencies.
   • Install OpenOOD v2.

2. Download Pretrained Checkpoints: Download teh checkpoints from OpenOOD repository (https://github.com/Jingkang50/OpenOOD)

   • The script assumes that the pretrained models are located in the checkpoints_dir directory. Make sure the checkpoints are downloaded.
   • The script uses the following checkpoints:
     • mnist_lenet_acc99.60.ckpt for Mnist
     • cifar10_resnet18_32x32_base_e100_lr0.1_default for Cifar-10
     • cifar100_resnet18_32x32_base_e100_lr0.1_default for Cifar-100
     • imagenet_res50_acc76.10.pth for ImageNet ResNet-50
     • vit_b_16-c867db91.pth for ImageNet ViT
     • swin_t-704ceda3.pth for ImageNet Swin
     • imagenet200_resnet18_224x224_base_e90_lr0.1_default for ImageNet-200 ResNet-18

3. Set Wandb API Key (Optional):

   • If you want to use Wandb, set the WANDB_API_KEY environment variable in the run.sh script.

4. Run the script:

   bash run.sh
   

The script will run GROOD on CIFAR-10 (i=1). To run on a different dataset, change the value of the variable i in the run.sh script.

By default, this runs CIFAR-10 (i=1). Change i in the script to select dataset:

• i=0 → MNIST
• i=1 → CIFAR-10
• i=2 → CIFAR-100
• i=3 → ImageNet-200
• i=4 → ImageNet-1K

---

Results

Main Results (AUROC, OpenOOD v1.5)

Method	CIFAR-10 Near	CIFAR-10 Far	CIFAR-100 Near	CIFAR-100 Far	IN-200 Near	IN-200 Far	IN-1K Near	IN-1K Far
MSP	88.0	90.7	80.3	77.8	83.3	90.1	76.0	85.2
ODIN	82.9	87.9	79.9	79.3	80.2	91.7	74.7	89.5
ReAct	87.1	90.4	80.7	80.4	81.9	92.3	77.4	93.7
KNN	90.6	93.0	80.2	82.4	81.6	93.2	71.1	90.2
VIM	88.7	93.5	75.0	81.7	78.7	91.3	72.1	92.7
GEN	88.2	91.4	81.3	79.7	82.9	91.4	76.9	89.8
NCI	88.8	91.3	81.0	81.3	83.5	93.7	78.6	95.5
GROOD (Ours)	91.2	93.8	78.9	84.4	83.4	92.2	78.9	94.8

---

Citation

If you find this work useful, please cite:

@article{elaraby2023grood,
  title={GROOD: GRadient-aware Out-Of-Distribution detection in interpolated manifolds},
  author={ElAraby, Mostafa and Sahoo, Sabyasachi and Pequignot, Yann and Novello, Paul and Paull, Liam},
  journal={arXiv preprint arXiv:2312.14427},
  year={2023}
}

---

Acknowledgment

This codebase builds upon OpenOOD.