TRAIN_AND_VALIDATE.md

March 18, 2025 · View on GitHub

VLM/LLM checkpoint preparation

victim model

You should first download the following checkpoint of viction model from huggleface website and save them in ./modellib dir.

LLaVA: LLaVA-1.5-13b
MiniGPT-v2: MiniGPT-v2 checkpoint and Llama-2-7b-chat-hf
- Note: refer to the readme file in Github to down the checkpoint for MiniGPT-v2.
- Please change the config in minigpt4/configs/models/minigpt_v2.yaml and eval_config/minigptv2_eval.yaml
CogVLM: CogVLM-chat-hf and Vicuna-7b-v1.5

Training

QR Attack

Run this script to train our AdaShiled
```
   bash scripts/train_ours_qr.sh
```
FigStep Attack

Run this script to train our AdaShiled
```
    bash scripts/train_ours_figstep.sh
```

Inference

Structure-based Jailbreak Attack

QR Attack

Run this script to evaluate our AdaShiled
```
   bash scripts/infer_ours_qr.sh
```
FigStep Attack

Run this script to evaluate our AdaShiled
```
    bash scripts/infer_ours_figstep.sh
```

Benign dataset ( MM-Vet ) Test

Extract mm-vet.zip to data/mmvet.
Run this script to evaluate our AdaShiled (To do)
```
  bash  scripts/infer_ours_benign.sh
```