Awesome Efficient Inference for Large Vision-Language Models

April 8, 2026 · View on GitHub

Jun Zhang*^1,2, Yicheng Ji*^1,2, Feiyang Ren*^1,2, Yihang Li*^1,2, Bowen Zeng*^1,2, Zonghao Chen*^1,2, Ke Chen^1,2, Lidan Shou^1,2, Gang Chen¹, Huan Li^1,2 (* equal contribution)

¹The State Key Laboratory of Blockchain and Data Security, Zhejiang University

²Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security

A curated list of papers, benchmarks, and resources for efficient inference of Large Vision-Language Models (LVLMs).
This repository accompanies our survey paper: "Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects"

📖 Introduction

Figure 1: LVLM Inference Pipeline and Encoding Stage Techniques. The figure illustrates the three-stage inference workflow (left) and detailed encoding stage optimization techniques (right), showing how visual information flows from raw input to the language model.

Large Vision-Language Models (LVLMs) enable complex reasoning over fine-grained visual inputs and long videos, yet their inference remains a primary bottleneck. This overhead is shaped not only by compute but by memory traffic, cache locality, and sequence length.

This repository provides a systematic taxonomy of efficiency techniques along three execution stages:

👁️ Encoding: Distilling visual information in compute-bound encoders
⚡ Prefilling: Mitigating quadratic attention via token compression and structured sparsity
⏩ Decoding: Overcoming the "visual memory wall" via KV cache compression, retrieval, and speculative execution

🧩 Stage-Wise Taxonomy

Figure 2: Taxonomy of Efficient Inference Techniques for LVLMs. We organize existing methods by the three stages of the inference lifecycle. Within each stage, techniques are further categorized by their specific optimization mechanisms to facilitate a clear understanding of WHERE and HOW computational redundancy is reduced.

📑 Table of Contents

Encoding Stage
Prefilling Stage
- Token Compression
- Sparse Attention
Decoding Stage
Benchmarks and Datasets
Survey and Related Work
Citation

👁️ Encoding Stage

Optimization techniques targeted at the encoding stage to reduce visual token count and encoding time.

Efficient Vision Encoders

Paper	Venue	Key Contribution
FastViT: A fast hybrid vision transformer using structural reparameterization	ICCV 2023	Novel token mixing operators and structural reparameterization
ConvLLaVA: Hierarchical backbones as visual encoder for large multimodal models	arXiv 2024	Compresses high-resolution images into information-rich visual features
FastVLM: Efficient vision encoding for vision language models	CVPR 2025	Hybrid vision encoder outputting fewer tokens and reducing encoding time
Glyph: Scaling Context Windows via Visual-Text Compression	arXiv 2025	DeepEncoder maintaining low activations under high-resolution input

Paper	Venue	Code	Key Contribution
Qwen2-VL: Enhancing vision-language model's perception of the world at any resolution	arXiv 2024		Native Dynamic Resolution framework enabling adaptive visual token generation
Video-ChatGPT: Towards detailed video understanding via large vision and language models	ACL 2024		Applies pooling over visual tokens to obtain compact visual representations
MovieChat: From dense token to sparse memory for long video understanding	CVPR 2024		Vision encoder explicitly trained for long video scenarios
Long context transfer from language to vision	arXiv 2024	-	Vision encoder explicitly trained for long video scenarios
LongVLM: Efficient long video understanding via large language models	ECCV 2024		Vision encoder explicitly trained for long video scenarios
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding	ICML 2025		Preserves full features for query-relevant frames while applying spatial pooling

Efficient Modality Adapters

Paper	Venue	Code	Key Contribution
BLIP-2: Bootstrapping Language-Image Pre-training	ICML 2023		Bridges modality gap with lightweight Querying Transformer (Q-Former)
Video-LLaMA: An instruction-tuned audio-visual language model	arXiv 2023		Proposes Video Q-Former for multi-modality video comprehension
Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM	arXiv 2024	-	Dynamic visual token compression architecture adapting to different lengths
TokenPacker: Efficient Visual Projector for Multimodal LLM	IJCV 2025	-	Coarse-to-fine scheme injecting enriched characteristics

Keyframe Selection

Training-Free

Paper	Venue	Code	Key Contribution
SeViLA: Self-chained image-language model for video localization	NeurIPS 2023		Uses frozen models as plug-and-play selectors for frame localization
KeyVideoLLM: Towards large-scale video keyframe selection	arXiv 2024	-	Employs frozen models as plug-and-play selectors for keyframe localization
Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation	arXiv 2025	-	Text-image matching network with Gumbel-Max trick
VideoTree: Adaptive tree-based video representation	CVPR 2025	-	Multi-granularity tree-based representation extracting query-relevant details
FOCUS: Efficient Keyframe Selection for Long Video Understanding	arXiv 2025	-	Formulates keyframe selection as combinatorial pure-exploration

Training-Aware

Paper	Venue	Code	Key Contribution
VILA: Efficient video-language alignment for video question answering	ECCV 2024	-	Text-guided Frame-Prompter learning to extract question-related frames
Frame-Voyager: Learning to query frames for video LLMs	arXiv 2024	-	Learns to query informative frame combinations
M-LLM based video frame selection for efficient video understanding	CVPR 2025	-	Uses spatial and temporal signals as supervision to train frame selector

Adaptive Resolution

Paper	Venue	Code	Key Contribution
VisionThink: Smart and efficient vision language model via reinforcement learning	arXiv 2025	-	Dynamically processes distinct samples with different resolutions
ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolution	arXiv 2025	-	Multiple MLP connectors with different compression ratios
Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation	arXiv 2025	-	Text-image matching network with Gumbel-Max trick
LongVU: Spatiotemporal Adaptive Compression	ICML 2025		Preserves full features for query-relevant frames

Encoding-Oriented Token Compression

Attention-Free

Paper	Venue	Code	Key Contribution
LLaVA-PruMerge: Adaptive Token Reduction	arXiv 2024		Reduces visual tokens according to similarities between class and spatial tokens
PVC: Progressive Visual Token Compression	arXiv 2024	-	Progressive compression strategy extending images as static videos
Less is More: A Simple yet Effective Token Reduction Method	arXiv 2024	-	Token reduction using both CLIP metric and similarity (TRIM)
FOLDER: Accelerating Multi-modal Large Language Models	arXiv 2025	-	Plug-and-play module in final vision backbone blocks for merging operations
Dynamic-VLM: Simple Dynamic Visual Token Compression	arXiv 2024	-	Dynamic visual token compression architecture adapting to different lengths
Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning	arXiv 2025	-	Selects informative tokens using visual attention (VisPruner)

Attention-Aware

Paper	Venue	Code	Key Contribution
VisionZip: Longer is Better but Not Necessary	arXiv 2024		Selects informative tokens using visual attention from encoder
HIVTP: Hierarchical Visual Token Pruning	arXiv 2025	-	Attention maps from middle encoder layers to estimate visual token importance
ToSA: Token Merging with Spatial Awareness	arXiv 2025	-	Token merging combining semantic and spatial awareness
SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference	ICCV 2025	-	Estimates token importance from visual encoder's self-attention maps

⚡ Prefilling Stage

Techniques to reduce computational and memory overhead during the prefilling stage.

Token Compression

Diversity-Guided

Paper	Venue	Code	Key Contribution
FrameFusion: Combining similarity and importance	arXiv 2024	-	Merges tokens in shallow layers and prunes in deep layers
G-Prune: Training-free visual token pruning from graph perspective	AAAI 2025	-	Similarity graph and information flow to retain representative tokens
DART: Stop looking for important tokens, duplication matters more	arXiv 2025	-	Pivot-based duplication pruning selecting tokens with low duplication
AIM: Adaptive Inference of Multi-Modal LLMs	arXiv 2024	-	Spatiotemporal token merging to reduce video redundancy
DivPrune: Diversity-based visual token pruning	CVPR 2025	-	Max-Min diversity optimization for token subset selection
FastVID: Dynamic density pruning for fast video LLMs	arXiv 2025		Temporal segmentation and density spatiotemporal pruning
DyCoke: Dynamic Compression of Tokens for Fast Video LLMs	CVPR 2025	-	Plug-and-play temporal compression module minimizing temporal redundancy
CDPruner: Maximizing Conditional Diversity for Token Pruning	arXiv 2025	-	Determinantal Point Processes (DPP) maximizing conditional diversity
PruneVid: Visual token pruning for efficient video LLMs	ACL 2025	-	Spatiotemporal token merging before LLMs
HoliTom: Holistic Token Merging for Fast Video LLMs	arXiv 2025	-	Global redundancy-aware segmentation followed by spatiotemporal merging
VidCom2: Video Compression Commander	arXiv 2025	-	Dynamic compression based on frame uniqueness
STTM: Multi-granular spatio-temporal token merging	ICCV 2025	-	Quadtree spatial transformation with directed pairwise merging
StreamingTOM: Streaming Token Compression	arXiv 2025	-	Causal temporal reduction with fixed per-frame budget
Dynamic-VLM: Simple Dynamic Visual Token Compression	arXiv 2024	-	Dynamic visual token compression architecture adapting to different lengths
TimeChat-Online: 80% Visual Tokens are Naturally Redundant	arXiv 2025	-	Differential token drop module filtering redundant content in streaming videos

Attention-Guided

Paper	Venue	Code	Key Contribution
FastV: An image is worth 1/2 tokens after layer 2	ECCV 2024		Learns attention patterns in early layers to prune visual tokens in deeper layers
PyramidDrop: Accelerating via pyramid visual redundancy reduction	arXiv 2024		Multi-stage pruning using attention score ranking
FrameFusion: Combining similarity and importance	arXiv 2024	-	Merges tokens in shallow layers and prunes in deep layers
SparseVLM: Visual token sparsification for efficient inference	arXiv 2024	-	Sparsifies visual tokens based on question prompt through text-visual attention scores
BTP: Balanced Token Pruning	arXiv 2025	-	Multi-stage pruning with diversity and attention ranking objectives
Fit and Prune: Fast and training-free visual token pruning	AAAI 2025	-	Minimizes divergence of attention distributions before and after pruning
ATP-LLaVA: Adaptive token pruning for LVLMs	CVPR 2025	-	Learnable adaptive token pruning module computing importance score
Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification	arXiv 2025	-	Chunk-based attention with historical context for video processing
StreamingVLM: Real-time understanding for infinite video streams	arXiv 2025	-	Maintains compact token subset by reusing attention sinks and recent token windows

Sparse Attention

Paper	Venue	Code	Key Contribution
VideoNSA: Native Sparse Attention Scales Video Understanding	arXiv 2025	-	End-to-end training with sparse attention preserving dense attention for text
SpargeAttn: Accurate sparse attention accelerating any model inference	arXiv 2025	-	Two-stage online filter to skip unimportant regions in sparse attention
XAttention: Block sparse attention with antidiagonal scoring	arXiv 2025	-	Block sparse attention with antidiagonal scoring for efficient block estimation
MMInference: Modality-Aware Permutation Sparse Attention	arXiv 2025	-	Identifies three distinct attention patterns in LVLMs with modality-aware permutation

⏩ Decoding Stage

Optimization techniques for the autoregressive decoding stage.

KV Cache Compression

Token-Level

Paper	Venue	Code	Key Contribution
LOOK-M: Look-once optimization in KV cache	arXiv 2024	-	Text-prior compression policy prioritizing textual KVs while evicting visual tokens
Elastic Cache: Efficient inference of vision instruction-following models	ECCV 2024	-	Cache merging strategy fusing less important KVs guided by distinct metrics
ReKV: Streaming video QA with in-context video KV-cache retrieval	arXiv 2025	-	Retrieval-based framework offloading video chunks to external memory
LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache	arXiv 2025	-	Dual-memory approach with short-term sliding window and compressed long-term memory
FastCache: Optimizing multimodal LLM serving	arXiv 2025	-	Lightweight modality-specific compressor learning compression patterns

Layer-Level

Paper	Venue	Code	Key Contribution
VL-Cache: Sparsity and modality-aware KV cache compression	arXiv 2024	-	Dynamically sets each layer's cache size according to measured attention sparsity
Meda: Dynamic KV cache allocation for efficient multimodal inference	arXiv 2025	-	Cross-modal attention entropy guiding cache allocation to layers with complex interactions
ST3: Accelerating MLLM by spatial-temporal visual token trimming	AAAI 2025	-	Progressive pruning of visual tokens in deeper layers based on decreasing visual importance
MadaKV: Adaptive Modality-Perception KV Cache Eviction	ACL 2025	-	Inter-layer compensation mechanism dynamically adjusting budgets
InfiniPot-V: Memory-Constrained KV Cache Compression	arXiv 2025	-	Layer-wise adaptive pooling with varying kernel sizes to balance abstraction and detail

Head-Level

Paper	Venue	Code	Key Contribution
SparseMM: Head Sparsity Emerges from Visual Concept Responses	arXiv 2025	-	Identifies vital visual heads and allocates asymmetric budgets based on visual relevance

Bit-Level

Paper	Venue	Code	Key Contribution
AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization	arXiv 2025	-	Adaptive mixed-precision quantization with high bit-width for critical tokens and 2-bit for others
CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMs	arXiv 2025	-	Channel-wise 1-bit quantization with post-calibration for extreme values
VidKV: Plug-and-Play 1.x-Bit KV Cache Quantization	arXiv 2025	-	Sub-2-bit quantization with differential treatment for K and V

Speculative Decoding

Training-Free

Paper	Venue	Code	Key Contribution
SpecVLM: Enhancing speculative decoding via verifier-guided token pruning	EMNLP 2025	-	Verifier-guided staged pruning removing up to 90% of vision tokens from draft model input

Training-Aware

Paper	Venue	Code	Key Contribution
Spec-LLaVA: Accelerating VLMs with Dynamic Tree-Based Speculative Decoding	arXiv 2025	-	Compact distilled draft model paired with tree-based verification algorithm
MSD: Speculative Decoding Reimagined for Multimodal Large Language Models	arXiv 2025	-	Two-stage training enabling draft model to acquire language modeling and visual perception
ViSpec: Accelerating VLMs with Vision-Aware Speculative Decoding	arXiv 2025	-	Lightweight vision adaptor to compress image tokens for draft model
FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference	arXiv 2025	-	Imitation-based draft model learning from deeper representations (self-speculative decoding)
Glyph: Scaling Context Windows via Visual-Text Compression	arXiv 2025		Introduces DeepEncoder maintaining low activations under high-resolution input
SpecVLM: Fast Speculative Decoding in Vision-Language Models	arXiv 2025	-	Elastic visual compressor adaptively selecting from multiple compression primitives
FLASH: Latent-Aware Semi-Autoregressive Speculative Decoding	arXiv 2025	-	Visual token compression mechanism and semi-autoregressive head for draft model optimization

Efficient Reasoning

Paper	Venue	Code	Key Contribution
Adaptive Fast-and-Slow Visual Program Reasoning for Long-Form VideoQA	arXiv 2025	-	Fast-slow reasoning framework routing simple queries to VideoLLM and complex ones to visual program workflow
PixelThink: Towards Efficient Chain-of-Pixel Reasoning	arXiv 2025	-	Reinforcement learning to regulate reasoning chain length based on task difficulty and model confidence
Prolonged reasoning is not all you need: Certainty-based adaptive routing	arXiv 2025	-	Certainty-based routing triggering long thought chains only when initial answer exhibits high uncertainty

📊 Benchmarks and Datasets

Multimodal Understanding Benchmarks

Benchmark	Description	Resources
MME	Comprehensive evaluation for multimodal LLMs	[Paper] [Leaderboard]
SEED-Bench	Benchmarking multimodal LLMs	[Paper] [Code]
MMMU	Massive multi-discipline multimodal understanding	[Paper] [Website]

Video Understanding Benchmarks

Benchmark	Description	Resources
Video-MME	First comprehensive video analysis benchmark	[Paper] [Website]
LongVideoBench	Long-context interleaved video-language understanding	[Paper] [Code]
MVBench	Comprehensive multi-modal video understanding	[Paper] [Code]

Survey	Description	Year
Token Compression Survey	Survey on token compression in LLMs	2025
Efficient LLMs Survey	Comprehensive survey on efficient LLMs	2024
MLLM Survey	Survey on multimodal LLMs	2024

📝 Citation

If you find this repository useful, please consider citing our survey paper:

@misc{zhang2026efficientinferencelargevisionlanguage,
      title={Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects}, 
      author={Jun Zhang and Yicheng Ji and Feiyang Ren and Yihang Li and Bowen Zeng and Zonghao Chen and Ke Chen and Lidan Shou and Gang Chen and Huan Li},
      year={2026},
      eprint={2604.05546},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.05546}, 
}

🤝 Contributing

We welcome contributions! If you find a relevant paper or resource that should be included, please:

Fork this repository
Add the paper to the appropriate category
Submit a pull request

For detailed guidelines, see CONTRIBUTING.md.

Disclaimer: This is a living document and will be continuously updated. If you notice any missing papers or have suggestions for better categorization, feel free to open an issue or submit a pull request.

Last Updated: 2026-04-08