Occ-VLM: Occupancy-Enhanced Vision-Language Model for 3D Scene Understanding

---

Overview

Occ-VLM extends Video-3D LLM with explicit 3D geometry priors. By integrating a SigLip-based occupancy prediction network as the vision tower, the model lifts 2D multi-view observations into dense 3D voxel features. World position embeddings inject spatial context, while a grounding head enables fine-grained 3D object localization — all unified within a single VLM.

---

News

• [2025-6] Code & model released. — Model.

Supported Tasks

Task	Type
ScanRefer	3D Visual Grounding
Multi3DRefer	Multi-Object 3D Grounding
SQA3D	3D Situated QA
ScanQA	3D Scene QA
Scan2Cap	3D Dense Captioning

Installation

git clone https://github.com/LaVi-Lab/Occ-VLM.git
cd Occ-VLM
conda create -n occvlm python=3.10 -y && conda activate occvlm
pip install --upgrade pip
pip install -e ".[train]"
pip install flash-attn --no-build-isolation

Data Preparation

See data preprocessing guide for full instructions. Quick steps:

1. Download ScanNet v2
2. Download EmbodiedScan
3. Extract images & point clouds via provided scripts
4. Process downstream annotations (ScanRefer, SQA3D, ScanQA, etc.)

Evaluation

Evaluate on each downstream task with the corresponding script:

# Usage: sh scripts/3d/eval/eval_<task>.sh <CKPT_NAME> <SAMPLING> <MAX_FRAMES>
#
# Parameters:
#   CKPT_NAME:    Checkpoint name under ./ckpt/ (e.g., occ3dllm-LLaVA-Qwen-video-mm-only-v2)
#   SAMPLING:     Frame sampling strategy — uniform | ratio90 | ratio95
#     - uniform:  Uniform sampling (equally spaced frames)
#     - ratio90:  Adaptive sampling until ≥90% scene voxel coverage
#     - ratio95:  Adaptive sampling until ≥95% scene voxel coverage
#   MAX_FRAMES:   Upper limit on sampled frames (e.g., 32)

# 3D Visual Grounding
sh scripts/3d/eval/eval_scanrefer.sh $CKPT $SAMPLING $MAX_FRAMES
sh scripts/3d/eval/eval_multi3drefer.sh $CKPT $SAMPLING $MAX_FRAMES

# 3D QA
sh scripts/3d/eval/eval_sqa3d.sh $CKPT $SAMPLING $MAX_FRAMES
sh scripts/3d/eval/eval_scanqa.sh $CKPT $SAMPLING $MAX_FRAMES

# 3D Dense Captioning
sh scripts/3d/eval/eval_scan2cap.sh $CKPT $SAMPLING $MAX_FRAMES

Use _lora variants for LoRA checkpoints (e.g., eval_scanrefer_lora.sh).

Acknowledgements

• LLaVA-Next — base codebase
• Video-3D LLM — multi-view video representation
• EmbodiedScan — occupancy network & preprocessing
• ScanNet, ScanRefer, Multi3DRefer, SQA3D, ScanQA — datasets