TrojViT: Trojan Insertion in Vision Transformers

Mengxin Zheng, Qian Lou, Lei Jiang

Accepted at CVPR 2023 [Paper Link].

Overview

• We propose a new attack framework, TrojViT, to breach the security of ViTs by creating a novel, stealthy, and practical ViT-specific backdoor attack TrojViT.

• We evaluate TrojViT on vit, deit and swin transformer.

Code Usage

Our codes support the TrojViT attack on SOTA Vision Transformers (e.g., DeiT-T, DeiT-S, and DeiT-B) on ImageNet validation dataset.

Key parameters

--data_dir: Path to the ImageNet folder.

--dataset_size: Evaluate on a part of the whole dataset.

--patch_select: Select patches based on the saliency map, attention map, or random selection.

--num_patch: Number of perturbed patches.

--sparse_pixel_num: Total number of perturbed pixels in the whole image.

--attack_mode: Optimize TrojViT based on the final cross-entropy loss only, or consider both cross-entropy loss and the attention map.

--attn_select: Select patches based on which attention layer.

Citation

@inproceedings{zheng2023trojvit,
  title={Trojvit: Trojan insertion in vision transformers},
  author={Zheng, Mengxin and Lou, Qian and Jiang, Lei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={4025--4034},
  year={2023}
}