Fast3D

Fast3D is a plug-and-play visual token pruning framework for accelerating 3D MLLMs (e.g., Chat-Scene). It could reach 90% visaul token pruning ratio with negligible performance drop through two technical innovations: (1) Global Attention Prediction (GAP), where a lightweight neural network is trained to predict the aggregated attention map from all layers of the target model, enabling efficient token importance estimation for precise pruning guidance; and (2) Sample-Adaptive visual token Pruning (SAP), which dynamically adjusts token budgets based on input complexity to achieve improved overall accuracy-efficiency trade-offs.

Performance Comparison

* w/ method x% indicates results with a x% average visual token pruning ratio.

Preparation

• Prepare the environment:

wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
conda create -n fast3d python=3.9.17
conda activate fast3d
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
wget --quiet https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.0.post2/flash_attn-2.7.0.post2+cu12torch2.5cxx11abiFALSE-cp39-cp39-linux_x86_64.whl
pip install flash_attn-2.7.0.post2+cu12torch2.5cxx11abiFALSE-cp39-cp39-linux_x86_64.whl
cd transformers
pip install -e .

• Download LLM backbone:

  • We use Vicuna-7B v1.5 in our experiments, which can be downloaded from Hugging Face.

  • Change the llama_model_path in config.py to the path of vicuna-7b-v1.5.

• Annotations and extracted features:

  Please follow the instructions in preprocess.

• Download Chat-Scene's pretrained checkpoint:

  We provide the pretrained checkpoint in Google Drive. Download it from either Link 1 or Link 2.

Chat-Scene Vanilla Inference

• Modify eval.sh:

val_tag="scanrefer#scan2cap#scanqa#sqa3d#multi3dref"
evaluate=True
pretrained_path="/path/to/pretrained_model.pth"

• Run: bash scripts/eval.sh

Inference with FastV

• Modify batch_eval_fastv.sh:

val_tag="scanrefer#scan2cap#scanqa#sqa3d#multi3dref"
evaluate=True
pretrained_path="/path/to/pretrained_model.pth"
# batch eval pruning ratios: 90%, 65%, 35%
rank_list=(15 60 90) # keep from 300 visual tokens
Ks=(2 6 16) # from which layer of 32 layers

• Run: bash scripts/batch_eval_fastv.sh

Inference with Fast3D (GAP)

train_tag="scanrefer#scan2cap#scanqa#sqa3d#multi3dref"
val_tag="scanrefer#scan2cap#scanqa#sqa3d#multi3dref"
evaluate=True
pretrained_path="/path/to/pretrained_model.pth"

attn_maps_root: /path/to/infer_attn_maps(extracted_attn_maps)
train_tags: scanrefer#scan2cap#scanqa#sqa3d#multi3dref
val_tags: scanrefer#scan2cap#scanqa#sqa3d#multi3dref

accelerate config
cd fast3d
bash train_fast3d.sh

eval_only: True
pretrained_model_path: /path/to/checkpoint_best.pth
save_attn_maps: True
attn_maps_root: /path/to/infer_attn_maps(extracted_attn_maps)
val_tags: scanrefer#scan2cap#scanqa#sqa3d#multi3dref

accelerate config
cd fast3d
bash test_fast3d.sh

val_tag="scanrefer#scan2cap#scanqa#sqa3d#multi3dref"
evaluate=True
pretrained_path="/path/to/pretrained_model.pth"
use_fast_v=False
use_fast_v_oracle=True
use_external_attn_maps=True
use_a_map_ori=False
val_attn_maps_path="/path/to/predicted_attn_maps"
# batch eval pruning ratios: 90%, 65%, 35%
rank_list=(15 60 90)
Ks=(2 6 16)

Inference with Fast3D (GAP+SAP)

• manual search total attention score threshold alpha: modify search_alpha.py

  alpha = 0.21
  target_pruning_ratio = 90
  tolerance = 2
  pred_attn_maps_path = "/path/to/predicted_attn_maps"
  

  then run python tasks/search_alpha.py to check if the alpha is valid.

• Modify batch_eval_fast3d_pred_attn_adaptive.sh: (We provide the predicted attention maps in Google Drive)

  val_tag="scanrefer#scan2cap#scanqa#sqa3d#multi3dref"
  evaluate=True
  pretrained_path="/path/to/pretrained_model.pth"
  use_fast_v=False
  use_fast_v_oracle=True
  use_external_attn_maps=True
  use_a_map_ori=False
  val_attn_maps_path="/path/to/predicted_attn_maps"
  alpha_list=(0.21)
  Ks=(0)
  

• Run: bash scripts/batch_eval_fast3d_pred_attn_adaptive.sh

📄 Citation

If you find this project useful in your research, please consider cite:

@misc{huang2025fast3daccelerating3dmultimodal,
      title={Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding}, 
      author={Wencan Huang and Daizong Liu and Wei Hu},
      year={2025},
      eprint={2507.09334},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2507.09334}, 
}

😊 Acknowledgement

Thanks to the open source of the following projects: Chat-Scene, FastV, and Vil3dref