README.md

March 9, 2026 · View on GitHub

HCC-3D: Hierarchical Compensatory Compression for 98% 3D Token Reduction in Vision-Language Models

Liheng Zhang¹, Jin Wang¹, Hui Li², Bingfeng Zhang^1†, Weifeng Liu¹
¹ China University of Petroleum (East China) ² The Hong Kong Polytechnic University ^† Corresponding Author

AAAI 2026

📰 News

[2026/03] 🚀 Code and model checkpoints are released.
[2026/01] 🎉 HCC-3D is accepted by AAAI 2026.

✨ Highlights

🗜️ 98% token compression: HCC-3D reduces 3D visual tokens from 513 → 12 — the most aggressive compression among all existing 3D-VLMs.
🏆 State-of-the-art: Outperforms MiniGPT-3D by +1.04% on ModelNet40 and +1.00% on Objaverse classification.
⚡ 52% faster training and 20% faster inference vs. MiniGPT-3D, running on a single RTX 4090 (24G) in just 11.9 hours.
🔧 Plug-and-play: The HCC module is architecture-agnostic and generalizes across different 3D-VLM frameworks (demonstrated with GreenPLM).

🏗️ Method

architecture

HCC-3D employs a dual-path hierarchical compression strategy:

Global Structure Compression (GSC): ng=8 learnable spatial queries attend over all 513 input tokens via multi-head cross-attention, producing a compact global representation that preserves overall 3D geometry.
Adaptive Detail Mining (ADM): Identifies under-attended but semantically rich regions missed by GSC using a complementary scoring mechanism (attention coverage × MLP importance), selects the top-K=96 features, and recompresses them into 4 detail tokens — yielding 12 tokens total.

🚀 Getting Started

📦 Installation

git clone https://github.com/lihengzhang02/HCC-3D.git
cd HCC-3D
conda env create -f environment.yml
conda activate hcc_3d
bash env_install.sh

Tested environment: 1× RTX 4090 24GB / Ubuntu 20.04 / CUDA 11.8 / Python 3.9 / PyTorch 2.0.0

🗂️ Data Preparation

Download all data files (~78GB) from PointLLM HuggingFace Datasets.

# Merge and extract Objaverse point clouds
cat Objaverse_660K_8192_npy_split_a* > Objaverse_660K_8192_npy.tar.gz
tar -xvf Objaverse_660K_8192_npy.tar.gz

Organize as follows:

HCC-3D/data
├── anno_data/
│   ├── PointLLM_brief_description_660K.json
│   ├── PointLLM_brief_description_660K_filtered.json
│   ├── PointLLM_brief_description_val_200_GT.json
│   ├── PointLLM_complex_instruction_70K.json
│   ├── object_ids_660K.txt
│   └── val_object_ids_3000.txt
├── modelnet40_data/
│   └── modelnet40_test_8192pts_fps.dat
└── objaverse_data/
    ├── 00000054c36d44a2a483bdbff31d8edf_8192.npy
    └── ...

🤗 Model Weights

Download from HuggingFace and place under params_weight/:

params_weight/
├── HCC_3D_stage_3/           # Stage III checkpoint
├── HCC_3D_stage_4/           # Stage IV checkpoint
├── Phi_2/                    # LLM backbone (Phi-2, 2.7B)
├── pc_encoder/               # Point-BERT encoder
├── all-mpnet-base-v2/        # Captioning evaluation
└── sup-simcse-roberta-large/ # Captioning evaluation

🎯 Training

export PYTHONPATH=$PWD

# Stage I:
CUDA_VISIBLE_DEVICES=0 python train.py --cfg-path ./train_configs/HCC_3D/stage_1.yaml

# Stage II:
CUDA_VISIBLE_DEVICES=0 python train.py --cfg-path ./train_configs/HCC_3D/stage_2.yaml

# Stage III:
CUDA_VISIBLE_DEVICES=0 python train.py --cfg-path ./train_configs/HCC_3D/stage_3.yaml

# Stage IV:
CUDA_VISIBLE_DEVICES=0 python train.py --cfg-path ./train_configs/HCC_3D/stage_4.yaml

📊 Evaluation

Step 1 — Generate outputs

export PYTHONPATH=$PWD

# ModelNet40 classification (run for prompt_index 0 and 1)
CUDA_VISIBLE_DEVICES=0 python hcc3d/eval/eval_modelnet.py \
    --out_path ./output/test \
    --cfg-path ./eval_configs/benchmark_evaluation_paper.yaml \
    --prompt_index 0

# Objaverse classification (run for prompt_index 0 and 1)
CUDA_VISIBLE_DEVICES=0 python hcc3d/eval/eval_objaverse.py \
    --out_path ./output/test --task_type classification \
    --cfg-path ./eval_configs/benchmark_evaluation_paper.yaml \
    --prompt_index 0

# Objaverse captioning
CUDA_VISIBLE_DEVICES=0 python hcc3d/eval/eval_objaverse.py \
    --out_path ./output/test --task_type captioning \
    --cfg-path ./eval_configs/benchmark_evaluation_paper.yaml \
    --prompt_index 2

Step 2 — LLM-based evaluation with Qwen2-72B-Instruct (Recommended)

We follow GreenPLM and use Qwen2-72B-Instruct as the evaluator for cost-effective and reproducible results. Get your API key from Alibaba Cloud, or self-host locally following the Qwen2 repo.

export PYTHONPATH=$PWD
export DASHSCOPE_API_KEY=sk-xxx

# Objaverse classification
python hcc3d/eval/evaluator_qwen.py \
    --results_path ./output/test/Objaverse_classification_prompt0.json \
    --eval_type open-free-form-classification \
    --model_type qwen2-72b-instruct --parallel --num_workers 4

# ModelNet40 classification
python hcc3d/eval/evaluator_qwen.py \
    --results_path ./output/test/ModelNet_classification_prompt0.json \
    --eval_type modelnet-close-set-classification \
    --model_type qwen2-72b-instruct --parallel --num_workers 4

# Captioning
python hcc3d/eval/evaluator_qwen.py \
    --results_path ./output/test/Objaverse_captioning_prompt2.json \
    --eval_type object-captioning \
    --model_type qwen2-72b-instruct --parallel --num_workers 4

Step 3 — Traditional metrics (Sentence-BERT & SimCSE)

CUDA_VISIBLE_DEVICES=0 python hcc3d/eval/traditional_evaluator.py \
    --results_path ./output/test/Objaverse_captioning_prompt2.json

☑️ TODO

Release training code
Release evaluation code
Release model checkpoints

📖 Citation

@inproceedings{zhang2026hcc3d,
  title     = {HCC-3D: Hierarchical Compensatory Compression for 98\% 3D Token Reduction in Vision-Language Models},
  author    = {Zhang, Liheng and Wang, Jin and Li, Hui and Zhang, Bingfeng and Liu, Weifeng},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year      = {2026}
}

LLaVA-Mini: Extreme 2D vision token reduction (576 → 1) for image and video VLMs
GreenPLM: 3D data-efficient point-language understanding
MiniGPT-3D: Efficient 3D-LLM alignment with 2D priors
PointLLM: Empowering LLMs to understand point clouds
ShapeLLM: 3D object understanding for embodied interaction

🙏 Acknowledgements

We thank the authors of PointLLM, MiniGPT-3D, GreenPLM, and Point-BERT for their excellent open-source work that this project builds upon.

📜 License

This project is released under the CC BY-NC 4.0 License.