[CVPR2025] Beyond Single-Modal Boundary: Cross-Modal Anomaly Detection through Visual Prototype and Harmonization

This repository provides the official implementation of Beyond Single-Modal Boundary: Cross-Modal Anomaly Detection through Visual Prototype and Harmonization.

We study cross-modal anomaly detection, where a model is trained on known modalities and tested on unseen ones. To improve generalization, our method learns Transferable Visual Prototypes directly in the visual space and uses Prototype Harmonization to adaptively combine prototypes from different source modalities. For few-shot settings, Visual Discrepancy Inference further enhances detection by comparing query images with a few normal samples from the target modality.

Experiments on ten datasets across RGB, 3D, MRI/CT, and thermal modalities show that our method achieves strong zero-shot and few-shot anomaly detection performance.

Overview of the training process in our proposed cross-modal anomaly detection framework.

---

Environment Setup

The code runs on a server equipped with 10 NVIDIA GeForce RTX 3090 GPUs under the following environment:

• Python 3.10.0
• PyTorch 2.5.1 + CUDA 11.8
• torchvision 0.20.1 + CUDA 11.8
• NumPy 1.26.3
• OpenCV 4.10.0
• scikit-learn 1.5.2
• scikit-image 0.24.0
• pandas 2.2.3
• matplotlib 3.10.0
• tqdm 4.67.1
• einops 0.8.0

Data Preparation

Download the datasets

We use ten public datasets across four modalities for cross-modal anomaly detection, including RGB datasets (MVTec AD, VisA, BTAD, and MPDD), 3D datasets (MVTec 3D-AD and Eyecandies), MRI/CT datasets (Brain MRI and OCT2017), and thermal datasets (WTB and Leaves).

• RGB: MvTec AD, VisA, BTAD, MPDD
• 3D: MVTec 3D-AD, Eyecandies
• MRI/CT: Brain MRI, OCT2017
• Thermal: WTB, Leaves

Convert datasets to the MVTec-style structure

After downloading the datasets, reorganize them into the standard MVTec AD format. Each category should contain train, test, and ground_truth folders, where normal training images are placed in train/good, test images are placed in test/good or test/<defect_type>, and pixel-level masks are placed in ground_truth/<defect_type> if available.

<dataset_name>/ 
└── <category_name>/
  ├── train/good/ 
  ├── test/good/ 
  ├── test/<defect_type>/ 
  └── ground_truth/<defect_type>/ 
  ...

The json files for each dataset can be downloaded from json files. For few-shot evaluation, the normal support samples for 1, 2, 4, and 8-shot settings are randomly generated and keep fixed across different methods for fair comparison, and can be downloaded from few-shot data.

Run on Zero-Shot Setting

For zero-shot training and evaluation, simply run the corresponding bash script:

  bash train_zero_shot.sh

  bash test_zero_shot.sh

Results under zero-shot setting.

Run on Few-Shot Setting

For few-shot evaluation, simply run the corresponding bash script:

  bash test_few_shot.sh

Results under few-shot setting.

Citation

  @inproceedings{mao2025beyond,
  title={Beyond single-modal boundary: Cross-modal anomaly detection through visual prototype and harmonization},
  author={Mao, Kai and Wei, Ping and Lian, Yiyang and Wang, Yangyang and Zheng, Nanning},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={9964--9973},
  year={2025}
}