[CVPR 2025] Explaining Domain Shifts in Language: Concept erasing for Interpretable Image Classification

Introduction

Concept-based models can map black-box representations to human-understandable concepts, which makes the decision-making process more transparent and then allows users to understand the reason behind predictions. However, domain-specific concepts often impact the final predictions, which subsequently undermine the model generalization capabilities, and prevent the model from being used in high-stake applications. In this paper, we propose a novel Language-guided Concept-Erasing (LanCE) framework. In particular, we empirically demonstrate that pre-trained vision-language models (VLMs) can approximate distinct visual domain shifts via domain descriptors while prompting large Language Models (LLMs) can easily simulate a wide range of descriptors of unseen visual domains. Then, we introduce a novel plug-in domain descriptor orthogonality (DDO) regularizer to mitigate the impact of these domain-specific concepts on the final predictions. Notably, the DDO regularizer is agnostic to the design of conceptbased models and we integrate it into several prevailing models. Through evaluation of domain generalization on four standard benchmarks and three newly introduced benchmarks, we demonstrate that DDO can significantly improve the out-of-distribution (OOD) generalization over the previous state-of-the-art concept-based models.

@article{zeng2025explaining,
  title={Explaining Domain Shifts in Language: Concept erasing for Interpretable Image Classification},
  author={Zeng, Zequn and Su, Yudi and Sun, Jianqiao and Wen, Tiansheng and Zhang, Hao and Wang, Zhengjue and Chen, Bo and Liu, Hongwei and Ma, Jiawei},
  journal={arXiv preprint arXiv:2503.18483},
  year={2025}
}

@inproceedings{zeng2023conzic,
  title={Conzic: Controllable zero-shot image captioning by sampling-based polishing},
  author={Zeng, Zequn and Zhang, Hao and Lu, Ruiying and Wang, Dongsheng and Chen, Bo and Wang, Zhengjue},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
  pages={23465--23476},
  year={2023}
}

pip install -r requirements.txt

data
└── CUB
    ├── CUB_200_2011
    │   ├── images
    │   ├── 
    │   └── 
    ├── CUB-200-Painting
    │   ├── images
    │   ├── 
    │   └── 
    └── ...    

python main.py --dataset CUB --alpha 0  --class_avg_concept --CBM_type clip_cbm --wandb

python main.py --dataset CUB --alpha 1  --class_avg_concept --CBM_type clip_cbm --wandb

python main_zeroshot.py --dataset CUB   --class_avg_concept --prompt_type origin --wandb