PCA: Persistence-Aware Compression and Aggregation for Fast Video Large Language Models

Abstract: Despite advances in Video Large Language Models (VLLMs) that have displayed promising outcomes in video understanding, the redundancy in the long-duration frames remains a hindrance to efficient reasoning. This paper introduces a training-free Persistence-Aware Compression and Aggregation (PCA) method designed to preserve high-fidelity raw visual information before the encoding stage. PCA can be built on arbitrary VLLMs and consists of two modules: (1) A Dynamic Downsampling (DD) module that adaptively removes redundant frames by analyzing frame-wise similarity; (2) A Persistence-Aware Frame Enhancement (PAFE) module that enriches each selected keyframe by aggregating the temporal context of its neighbors, ensuring that essential information is preserved even after aggressive frame reduction. Our approach substantially reduces the computation of long-context modeling, while enhancing the performance of the baseline model. Extensive experiments demonstrate that PCA consistently outperforms existing state-of-the-art approaches in both efficiency and accuracy, achieving a speedup of 1.8X to 2.5X compared to the baseline VLLM. The code is available in the supplementary material.

Install

Install the inference package:

conda create -n PCA python=3.10 -y
conda activate PCA
pip install --upgrade pip  
pip install lmms-eval
pip install -e ".[train]"

Evaluation

• We use the lmms-eval toolkit to evaluate our models.

Test on the specified task：

accelerate launch --num_processes=1 \
-m lmms_eval \
--model llava_onevision \
--model_args pretrained=lmms-lab/llava-onevision-qwen2-7b-ov,conv_template=qwen_1_5,model_name=llava_qwen \
--tasks mvbench \ 
--batch_size 1 \
--log_samples \
--log_samples_suffix llava_onevision \
--output_path ./logs/

Results