Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding

---

🔍 Overview

Occ-VLM is an occupancy-grounded vision-language framework for 3D scene understanding using only posed RGB images. Unlike existing methods that rely on explicit 3D inputs or separate 3D geometry encoders, Occ-VLM reconstructs 3D occupancy as an auxiliary geometric prior and uses it to associate foreground 2D tokens with 3D space.

This design preserves the rich semantics learned from vision-language pre-training while introducing explicit 3D spatial awareness within a unified 2D vision encoder. Occ-VLM achieves state-of-the-art multi-view occupancy prediction and competitive performance on 3D visual question answering, dense captioning, and visual grounding benchmarks.

✨ Highlights

• Occupancy-enhanced 3D representation
  Lift multi-view image features into dense voxel-level 3D representations using a SigLip-based occupancy network.

• Occupancy-aware spatial encoding
  Inject explicit spatial context through world position embeddings.

• Unified 3D vision-language reasoning
  A single VLM supports grounding, question answering, and dense captioning tasks.

---

🔥 News

• [2025.06] 🚀 Code and model released. — Model

---

🎯 Supported Tasks

Task	Category	Dataset
ScanRefer	3D Visual Grounding	ScanNet
Multi3DRefer	Multi-Object 3D Grounding	ScanNet
SQA3D	3D Situated Question Answering	ScanNet
ScanQA	3D Scene Question Answering	ScanNet
Scan2Cap	3D Dense Captioning	ScanNet

---

🚀 Quick Start

1. Installation

Clone the repository and create the environment:

git clone https://github.com/LaVi-Lab/Occ-VLM.git
cd Occ-VLM

conda create -n occvlm python=3.10 -y
conda activate occvlm

pip install --upgrade pip
pip install -e ".[train]"
pip install flash-attn --no-build-isolation

2. Download the Checkpoint

The released model is available on Hugging Face:

👉 Electronics/occ3dllm-LLaVA-Qwen-video

Place the checkpoint under the repository checkpoint directory, for example:

Occ-VLM/
├── ckpt/
│   └── occ3dllm-LLaVA-Qwen-video-mm-only-v2/
├── scripts/
├── llava/
└── ...

3. Run Evaluation

General usage:

sh scripts/3d/eval/eval_<task>.sh <CKPT_NAME> <SAMPLING> <MAX_FRAMES>

Example:

sh scripts/3d/eval/eval_scanrefer.sh \
    occ3dllm-LLaVA-Qwen-video-mm-only-v2 \
    uniform \
    32

---

📦 Data Preparation

Occ-VLM currently supports evaluation on ScanNet-based 3D vision-language benchmarks.

Please refer to the full preprocessing instructions:

👉 scripts/3d/preprocessing/README.md

Required Data

1. Download ScanNet v2.
2. Download EmbodiedScan.
3. Extract RGB images and point clouds using the provided preprocessing scripts.
4. Process downstream task annotations, including ScanRefer, Multi3DRefer, SQA3D, ScanQA, and Scan2Cap.

A typical processed dataset layout may follow the structure below:

data/
├── scannet/
├── embodiedscan/
├── scanrefer/
├── multi3drefer/
├── sqa3d/
├── scanqa/
└── scan2cap/

Please follow the preprocessing guide for the exact paths and generated files expected by the codebase.

---

📊 Evaluation

Evaluate each downstream task using the corresponding script.

3D Visual Grounding

sh scripts/3d/eval/eval_scanrefer.sh $CKPT $SAMPLING $MAX_FRAMES
sh scripts/3d/eval/eval_multi3drefer.sh $CKPT $SAMPLING $MAX_FRAMES

3D Question Answering

sh scripts/3d/eval/eval_sqa3d.sh $CKPT $SAMPLING $MAX_FRAMES
sh scripts/3d/eval/eval_scanqa.sh $CKPT $SAMPLING $MAX_FRAMES

3D Dense Captioning

sh scripts/3d/eval/eval_scan2cap.sh $CKPT $SAMPLING $MAX_FRAMES

For LoRA checkpoints, use the corresponding _lora evaluation scripts, e.g.:

sh scripts/3d/eval/eval_scanrefer_lora.sh $CKPT $SAMPLING $MAX_FRAMES

---

🤗 Model Zoo

Model	Backbone	Checkpoint
Occ-VLM	LLaVA-Qwen-Video + Occupancy Vision Tower	Hugging Face

---

📁 Repository Structure

A simplified overview of the repository:

Occ-VLM/
├── ckpt/                    # Model checkpoints
├── scripts/
│   └── 3d/
│       ├── preprocessing/   # Data preprocessing
│       └── eval/            # Evaluation scripts
├── llava/                   # Core model implementation
├── README.md
└── ...

---

🙏 Acknowledgements

We sincerely thank the authors of the following projects and datasets for making their work publicly available:

• LLaVA-NeXT — base codebase
• Video-3D LLM — multi-view video representation
• EmbodiedScan — occupancy network and preprocessing
• ScanNet — 3D indoor scene dataset
• ScanRefer — 3D visual grounding
• Multi3DRefer — multi-object 3D grounding
• SQA3D — situated question answering
• ScanQA — 3D question answering

---

📝 Citation

If you find Occ-VLM useful for your research, please consider citing our work.

@article{li2026occ,
  title={Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding},
  author={Li, Jianing and Fang, Zhou and Liu, Yijiang and Du, Li},
  journal={arXiv preprint arXiv:2606.19776},
  year={2026}
}
@inproceedings{zheng2025video,
  title={Video-3d llm: Learning position-aware video representation for 3d scene understanding},
  author={Zheng, Duo and Huang, Shijia and Wang, Liwei},
  booktitle={2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={8995--9006},
  year={2025},
  organization={IEEE}
}
@inproceedings{wang2024embodiedscan,
  title={Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai},
  author={Wang, Tai and Mao, Xiaohan and Zhu, Chenming and Xu, Runsen and Lyu, Ruiyuan and Li, Peisen and Chen, Xiao and Zhang, Wenwei and Chen, Kai and Xue, Tianfan and others},
  booktitle={2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={19757--19767},
  year={2024},
  organization={IEEE}
}