SaFeR-VLM

Installation

Clone the repository and install dependencies:

cd SaFeR-VLM

uv venv safer-vlm --python 3.11 && source safer-vlm/bin/activate && uv pip install --upgrade pip

uv pip install --no-cache-dir "vllm==0.9.1" "torch==2.7.0" "torchvision==0.22.0" "torchaudio==2.7.0" tensordict torchdata \
    "transformers[hf_xet]>=4.51.0" accelerate datasets peft hf-transfer \
    "numpy<2.0.0" "pyarrow>=15.0.0" "grpcio>=1.62.1" "optree>=0.13.0" pandas \
    ray[default] codetiming hydra-core pylatexenc qwen-vl-utils wandb liger-kernel mathruler

ABI_FLAG=$(python -c "import torch; print('TRUE' if torch._C._GLIBCXX_USE_CXX11_ABI else 'FALSE')") && \
    URL="https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.0.post2/flash_attn-2.8.0.post2+cu12torch2.7cxx11abi${ABI_FLAG}-cp310-cp310-linux_x86_64.whl" && \
    wget -nv -P /opt/tiger "${URL}" && \
    pip install --no-cache-dir "/opt/tiger/$(basename ${URL})"

pip install -e .

pip install wandb or swanlab

or

💡 You can also use the Docker support from EasyR1 for quick setup:

• A Dockerfile is provided for building your own environment.
• A pre-built Docker image is available:

docker pull hiyouga/verl:ngc-th2.7.0-cu12.6-vllm0.9.1

---

Quick Start

1. Download pretrained model from huggingface

# Download Qwen2.5-VL-3B-Instruct
hfi download Qwen/Qwen2.5-VL-3B-Instruct --repo-type model

# Download Qwen2.5-VL-7B-Instruct
hfi download Qwen/Qwen2.5-VL-7B-Instruct --repo-type model

# Download GRM-RL-7B (reward model)
hfi download generative-rlhf-v/GRM-RL-7B --repo-type model

2. Download source dataset from huggingface

# Download SPA-VL
hf download sqrti/SPA-VL --repo-type dataset

# Download align-anything (using text-image-to-text subset)
hf download PKU-Alignment/align-anything --repo-type dataset

# Download BeaverTails-V
hf download saferlhf-v/BeaverTails-V --repo-type dataset

3. QI-Box Selection

python SaFeR-VLM\scripts\data_infer.py

python SaFeR-VLM\scripts\data_eval.py

python SaFeR-VLM\scripts\data_select.py

python SaFeR-VLM\scripts\data_transfer.py

4. Launch Reward Model Inference Service

Start the reward model with vLLM backend (make sure the reward model is downloaded first):

bash scripts\lauch_vllm.sh

5. Run RL Training

Start reinforcement learning training (here using safer-VLM-7B with GRM-RL-7B as the reward model and websocket backend):

bash examples\safer_vlm_7b_rm7b_ws.sh

---

Eval

1. Model Deployment

Refer to Launch Reward Model Inference Service.

2. Run Model Inference

Make sure the port matches the deployed service:

python eval/infer_mmsafety.py --model_name $model --port $port

3. Evaluation with GPT

Load your API key:

export OPENAI_API_KEY="your key"
export OPENAI_BASE_URL="https://openrouter.ai/api/v1"

Run the evaluation script:

python SaFeR-VLM/eval/eval_mmsafety.py  --model_name $model

⚠️ Note

If Hugging Face is blocked in your region, set the mirror before running:

export HF_ENDPOINT=https://hf-mirror.com