Dynamic Accumulated Attention Map

This repository contains source code necessary to reproduce the explanation maps for ViT model's prediction in image classification task in this paper Dynamic Accumulated Attention Map for Interpreting Evolution of Decision-making in Vision Transformer (Pattern Recognition, 2025). If you use the code in this paper, please consider citing:

@article{yiliaoPR2025dynamic,
  title={Dynamic Accumulated Attention Map for Interpreting Evolution of Decision-making in Vision Transformer},
  author={Liao, Yi and Gao, Yongsheng and Zhang, Weichuan},
  journal={Pattern Recognition},
  volume={165},
  pages={111607},
  year={2025},
  publisher={Elsevier}
}

Method Overview

To generate the attention flow for ViT model's prediction, an image is fed into a ViT model to obtain the decison-making [cls] token. During the calculation of the [cls] token, the semantic spatial map is stored by the proposed decomposition module. Then the semantic spactial map is combined linearly with the importance coefficients derived from the classification score, forming the attention map for a ViT block. According the residual structure in ViT, the attention flow is constructed by acculmulating attention maps from the first ViT block to the last ViT block. The method is depicted by the following,

The Environment Setup

To run the code successfully, please download the software Anaconda3 from the official website and install it in advance.

1. After installation, open a Command Prompt (Windows) or Terminal (Linux), input the following command,

conda create -n daam python==3.8

2. Please activate the enrionment daam by running

conda activate daam

3. Please install the pytorch framework and the below libraries by runing the following commands in sequence,

conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge
python -m pip install opencv-python -i https://pypi.tuna.tsinghua.edu.cn/simple
conda install -c anaconda pillow scikit-learn pandas seaborn cython dbf
conda install -c conda-forge h5py timm einops yacs cvxpy nested_dict

Dynamic Accumulated Attention Map for Supervised ViT

We provide an exmple of using our proposed DAAM algorithm to generate the attention flow for DeiT(ViT-Small-Patch16), please directly run the following command in Command Prompt (Windows) or Terminal (Linux),

python VIT_DynamicAccumulatedAttentionMap_oneSample.py

The pretrained model's weights are automatically downloaded into the pytorch cache by runing the above command.

Dynamic Accumulated Attention Map for Self-Supervised ViT

To generate the attention flow for self-supervised ViT models,

1. Download the pretrained model weights of the self-supervised ViT DINO and XCiT.

2. Generate the memory bank by following the algorithm in 2018 CVPR paper "Unsupervised Feature Learning via Non-parameteric Instance Discrimination" and using the official pytorch released code.

3. Generate the explantion maps for all ViT blocks by using the proposed DAAM.

For convenience, we provid an implementation example using DINO(ViT-Small-Patch8) to demonstrate how the proposed DAAM works. The pretrained model weights are located in the pretrainedmodels folder. The corresponding memory bank can be downloaded from this Google Drive. After downloading, please unzip them and then place the 4 files,

• "trainfeat.pth"

• "testfeat.pth"

• "trainlabels.pth"

• "testlabels.pth"

into the directory memorybank/DINO/ImageNet2012/dino_small_p8/. Finally, please run the following command in Command Prompt (Windows) or Terminal (Linux),

python DINO_DAAM_Visualization_oneSample.py