[ICCV2025 Highlight] Where, What, Why: Towards Explainable Driver Attention Prediction

This repository contain the official code and dataset for the paper:

"Where, What, Why: Towards Explainable Driver Attention Prediction"
Yuchen Zhou*, Jiayu Tang*, Xiaoyan Xiao, Yueyao Lin, Linkai Liu, Zipeng Guo, Hao Fei, Xiaobo Xia, Chao Gou†
(*Equal contribution, †Corresponding author)
📍 Sun Yat-sen University & National University of Singapore
📄 arXiv 2506.23088

---

🔍 Overview

This work introduces Explainable Driver Attention Prediction (EDAP) — a novel task that jointly predicts:

• 🧭 Where: pixel-level driver attention maps
• 🧩 What: semantic understanding of attended regions
• 🧠 Why: cognitive reasoning behind the attention

We present:

• W³DA: The first large-scale dataset with spatial, semantic, and causal annotations across normal, critical, and accident driving scenarios. It’s available here.
• LLada: A multimodal large language model (MLLM)-driven architecture that unifies pixel prediction, semantic parsing, and causal reasoning in an end-to-end framework.

---

🛠️ To-Do List

We are actively building this repository. The full code and cleaned dataset will be released progressively. Stay tuned!

• [✔] Initial Code Release
• Refactored & Cleaned Code
• [✔] Cleaned W³DA Dataset Release

---

Installation

pip install -r requirements.txt
sudo apt-get install openjdk-8-jdk

Training

Pre-trained weights

In LLada, we employ the pre-trained LLaVA weights LLaVA-Lightning-7B-v1-1 , which can be directly downloaded from this link. As to MLLM’s vision backbone, we adopt the clip-vit-large-patch14, which can obtained from link.

Data Preparation

W³DA Dataset sources from 4 public datasets, BDDA, DR(eye)VE, LBW, DADA-2000.

It is available here. Upon downloading the W³DA dataset, you need to organize it as follows:

├── dataset
│   ├── BDDA
│   │   ├── training
│   │       ├── 0001
│   │       	└── raw_frames
│   │       	└── gazemap_frames
│   │       	└── 000001.json
│   │       	└── 000002.json
│   │       	└── 000003.json
│   │       	└── ......
│   │       └── 0002
│   │       └── 0003
│   │       └── ......
│   │   └── validation
│   │   └── test
│   ├── DReyeVE
│   │   ├── training
│   │   └── validation
│   │   └── test
│   ├── LBW
│   │   ├── training
│   │   └── validation
│   │   └── test
│   ├── DADA
│   │   ├── training
│   │   └── validation
│   │   └── test

To train the model, please use the following instruction, where dataset designates the sub-dataset used for training; train_sample_rates specifies the sampling rate of training samples for each sub-dataset.

deepspeed --master_port=24999 train_ds.py \
  --version="PATH_TO_LLaVA" \
  --vision-tower="PATH_TO_CLIP" \
  --dataset_dir="PATH_TO_W3DA" \
  --log_base_dir="PATH_TO_LOG" \
  --dataset="BDDA||DReyeVE||LBW||DADA" \
  --train_sample_rates="8,5,2,7" \
  --exp_name="Attn-7b"

When training is finished, you can obtain the full model using the zero_to_fp32.py script which is automatically generated by deepspeed engine within the folder ./runs/Attn-7b/ckpt_model.

cd ./runs/Attn-7b/ckpt_model && python zero_to_fp32.py . ../pytorch_model.bin

Merge LoRA weight

Merge the LoRA weights of pytorch_model.bin, save the resulting model into your desired path in the Hugging Face format:

CUDA_VISIBLE_DEVICES="" python3 merge_lora_weights_and_save_hf_model.py \
  --version="PATH_TO_LLaVA" \
  --vision-tower="PATH_TO_CLIP" \
  --weight="Attn-7b/pytorch_model.bin" \
  --save_path="./ATTN-7B"

Validation

To validate the finetuned model, please use the following instruction, where val_dataset designates the sub-datasets to evaluate. eval_only is required to enable the evaluation process.

deepspeed --master_port=24999 train_ds.py \
  --version="PATH_TO_LLada" \
  --vision-tower="PATH_TO_CLIP" \
  --dataset_dir="PATH_TO_W3DA" \
  --log_base_dir="PATH_TO_LOG" \
  --val_dataset="BDDA||DReyeVE||LBW||DADA" \
  --val_batch_size=1 \
  --eval_only

📦 Citation

If you find this work helpful, please cite:

@inproceedings{zhou2025where,
      title={Where, What, Why: Towards Explainable Driver Attention Prediction}, 
      author={Yuchen Zhou and Jiayu Tang and Xiaoyan Xiao and Yueyao Lin and Linkai Liu and Zipeng Guo and Hao Fei and Xiaobo Xia and Chao Gou},
      year={2025},
      eprint={2506.23088},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2506.23088}, 
}

Acknowledgement

• This work is built upon the LLaVA and LISA.

📦 Contact

Feel free to reach out for questions or collaborations:

📧 Yuchen Zhou: zhouych37@mail2.sysu.edu.cn