Efficient Vision Language Models: A Survey

Paper - https://onlinelibrary.wiley.com/share/author/4PSWRRQC2WJVIMMAUHDY?target=10.1002/widm.70036

Gaurav Shinde, Anuradha Ravi, Emon Dey, Shadman Sakib, Milind Rampure, Nirmalya Roy

University of Maryland, Baltimore County (UMBC)

If you like our project, please give us a star ⭐ on GitHub.

🚀 Exciting news- Our survey has been accepted at WIREs DMKD (11.7 IF)

🤝 Support & Collaboration

We welcome feedback and contributions to improve this survey and repository. The repository will be actively maintained with emerging research. Feel free to reach out via email (gshinde1@umbc.edu) with any suggestions.

📌 Abstract

Vision-language models (VLMs) integrate visual and textual information, enabling a wide range of applications such as image captioning and visual question answering, making them crucial for modern AI systems. However, their high computational demands pose challenges for real-time applications. This has led to a growing focus on developing efficient visionlanguage models. In this survey, we review key techniques for optimizing VLMs on edge and resource-constrained devices. We also explore compact VLM architectures, frameworks and provide detailed insights into the performancelatency trade-offs of efficient VLMs. Our objective is to foster deeper research in this area.

📊 Evaluation of Compact VLMs

SmolVLM with 2.2B parameters consistently achieves the highest accuracy across eight benchmarks, followed by the 500M and then the 256M models. Interestingly, Idefics with 80B parameters yields the lowest performance across most benchmarks. In terms of throughput, the 256M model delivers the highest, followed by the 500M, 2.2B, and finally the 80B model. A higher throughput indicates better efficiency. These results highlight a critical insight: achieving optimal performance on edge or pervasive IoT devices requires a careful balance between accuracy and efficiency, tailored to the specific constraints of the deployment hardware, such as memory and GPU availability.

📖 Table of Content

• 🚀 Pre-deployment techniques
  • Quantization
    • Post-Training Quantization
    • Quantization-Aware Training
  • Low-rank Approximation
  • Pruning
    • Structured
    • Unstructured
  • Knowledge Distillation
  • Other Methods
• 🎯 Efficient Finetuning
  • Parameter Efficient
    • Low-Rank Adapters
    • Prompt Tuning
    • Adapter-based Methods
    • Prefix Tuning
  • Memory Efficient
• ⚡ Runtime Optimization
  • Token Reduction
  • Test-Time Adaption
    • Test-Time Augmentation
    • Test-Time Prompt Tuning
• 🔒🌐 Privacy-Preserving Distributed VLM

🚀 Pre-deployment Techniques

Quantization

Post-Training Quantization

• MBQ-Modality-Balanced Quantization for Large Vision-Language Models, arXiv, 2024 [Paper] [Code]
• Q-VLM- Post-training Quantization for Large Vision Language Models, NeurIPS, 2024 [Paper] [Code]
• P4Q- Learning to Prompt for Quantization in Visual-language Models, arXiv, 2024 [Paper]
• BiLLM- Pushing the Limit of Post-Training Quantization for LLMs, ICML, 2024 [Paper] [Code]
• NoisyQuant- Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers, CVPR, 2023 [Paper] [Code]
• LRQuant-Learnable and Robust Post-Training Quantization for Large Language Models, ACL, 2024 [Paper] [Code]
• PTQ4ViT- Post-Training Quantization for Vision Transformers with Twin Uniform Quantization, arXiv, 2024 [Paper] [Code]
• PTQ4SAM- Post-Training Quantization for Segment Anything, CVPR, 2024 [Paper] [Code]
• SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 QuantizationarXiv, 2024 [Paper] [Code]
• SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration arXiv, 2024 [Paper] [Code]
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning arXiv, 2023 [Paper] [Code]
• Post-Training Quantization for Vision Transformer,arXiv, 2021 [Paper]
• FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer,arXiv, 2021 [Paper] [Code]
• SmoothQuant: accurate and efficient post-training quantization for large language models,arXiv, 2022 [Paper] [Code]

Quantization-Aware Training

• Boost Vision Transformer with GPU-Friendly Sparsity and Quantization, CVPR, 2023 [Paper]
• Advancing Multimodal Large Language Models with Quantization-aware Scale Learning for Efficient Adaptation, ACM, 2024 [Paper] [Code]
• Q-ViT- Fully Differentiable Quantization for Vision Transformer, arXiv, 2022 [Paper]
• LLM-QAT-Data-Free Quantization Aware Training for Large Language Models, arXiv, 2024 [Paper] [Code]
• EfficientQAT- Efficient Quantization-Aware Training for Large Language Models, arXiv, 2024 [Paper] [Code]
• Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precision, arXiv, 2023 [Paper] [Code]
• PackQViT: faster sub-8-bit vision transformers via full and packed quantization on the mobile, ACM, 2023 [Paper] [Code]
• Fixed-point Quantization for Vision Transformer, IEEE, 2021 [Paper]
• QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language ModelsarXiv, 2023 [Paper] [Code]
• Low-Rank Quantization-Aware Training for LLMs arXiv, 2024 [Paper] [Code]

Low-rank Approximation

• SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximation, arXiv, 2024 [Paper] [Code]
• PELA: Learning Parameter-Efficient Models with Low-Rank Approximation, CVPR, 2024 [Paper] [Code]
• Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs, CVPR, 2024 [Paper]

Pruning

Structured

• SmartTrim- Adaptive Tokens and Attention Pruning for Efficient VLMs, arXiv, 2023 [Paper]
• OSSCAR- One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization, ICML, 2024 [Paper] [Code]
• MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric, CVPR, 2024 [Paper]
• Isomorphic Pruning for Vision Models, arXiv, 2024 [Paper] [Code]

Unstructured

• Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration, arXiv, 2024 [Paper] [Code]
• MULTIFLOW- Shifting Towards Task-Agnostic Vision-Language Pruning, CVPR, 2024 [Paper] [Code]
• ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models, ICLR, 2024 [Paper] [Code]

Knowledge Distillation

• VLDadaptor_Domain_Adaptive_Object_Detection_With_Vision-Language_Model_Distillation, IEEE TMM, 2024 [Paper] [Code]
• Building Vision-Language Models on Solid Foundations with Masked Distillation, CVPR, 2024 [Paper]
• PromptKD: Unsupervised Prompt Distillation for Vision-Language Models, CVPR, 2024 [Paper] [Code]
• KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation, arXiv, 2021 [Paper]
• Compressing Visual-linguistic Model via Knowledge Distillation, ICCV, 2021 [Paper]

Other Methods

• Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning, CVPR, 2017 [Paper]
• Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters, CVPR, 2024 [Paper] [Code]
• Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models, arXiv, 2024 [Paper] [Code]
• Scaling Vision-Language Models with Sparse Mixture of Experts, arXiv, 2023 [Paper]

🎯 Efficient Finetuning

Parameter Efficient

Low-Rank Adapters

• Low-Rank Few-Shot Adaptation of Vision-Language Models , arXiv, 2024 [Paper] [Code]
• Advancing Vision-Language Models with Adapter Ensemble Strategies , EMNLP, 2024 [Paper]
• Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation, arXiv, 2024 [Paper]
• Complementary Subspace Low-Rank Adaptation of Vision-Language Models for Few-Shot Classification, arXiv, 2025 [Paper]
• One Head Eight Arms: Block Matrix based Low Rank Adaptation for CLIP-based Few-Shot Learning, arXiv, 2025 [Paper]
• Adaptive Rank, Reduced Forgetting: Knowledge Retention in Continual Learning Vision-Language Models with Dynamic Rank-Selective LoRA, arXiv, 2024 [Paper] [Code]
• LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models, arXiv, 2025 [Paper]
• Serial Low-rank Adaptation of Vision Transformer, arXiv, 2025 [Paper]

Prompt Tuning

• Visual-Language Prompt Tuning with Knowledge-guided Context Optimization, CVPR, 2023 [Paper] [Code]
• Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model,IEEE TMM, 2023 [Paper] [Code]
• Distribution-Aware Prompt Tuning for Vision-Language Models,ICCV, 2023 [Paper] [Code]

Adapter-based Methods

• MMA: Multi-Modal Adapter for Vision-Language Models,CVPR, 2024 [Paper] [Code]
• VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks,CVPR, 2022 [Paper] [Code]
• Meta-Adapter: An Online Few-shot Learner for Vision-Language Mode,NeurIPS, 2023 [Paper][Code]

Prefix Tuning

• Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models,arXiv, 2023 [Paper] [Code]
• User-Aware Prefix-Tuning is a Good Learner for Personalized Image Captioning,arXiv, 2024 [Paper]
• Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning,arXiv, 2024 [Paper]

Memory Efficient

• Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning, arXiv, 2023 [Paper] [Code]
• Time-, Memory- and Parameter-Efficient Visual Adaptation, CVPR, 2024 [Paper]
• M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension, arXiv, 2024 [Paper] [Code]
• SLIMFIT- Memory-Efficient Fine-Tuning of Transformer-based Models using Training Dynamics, arXiv, 2023 [Paper] [Code]

⚡ Runtime Optimization

Token Reduction

• Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Models, arXiv, 2023 [Paper]
• PuMer: Pruning and Merging Tokens for Efficient Vision Language Models, ACL 2023 [Paper] [Code]

Test-Time Adaption

• Frustratingly Easy Test-Time Adaptation of Vision-Language Models, NeurIPS, 2024 [Paper] [Code]
• Efficient Test-Time Adaptation of Vision-Language Models, CVPR, 2024 [Paper] [Code]
• SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models, NeurIPS, 2023 [Paper]
• Online Gaussian Test-Time Adaptation of Vision-Language Models, arXiv 2025 [Paper] [Code]

Test-Time Augmentation

• TextAug: Test time Text Augmentation for Multimodal Person Re-identification, arXiv, 2023 [Paper]
• On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?, CVPR, 2024 [Paper] [Code]

Test-Time Prompt Tuning

• Efficient Test-Time Prompt Tuning for Vision-Language Models, arXiv, 2024 [Paper]
• C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion, ICLR, 2024 [Paper] [Code]
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models, NeurIPS, 2022 [Paper] [Code]

🔒🌐 Privacy-Preserving Distributed VLM

• pFedPrompt: Learning Personalized Prompt for Vision-Language Models in Federated Learning, ACM WWW, 2023 [Paper]
• Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method, NeurIPS, 2024 [Paper] [Code]
• Efficient Adapting for Vision-language Foundation Model in Edge Computing Based on Personalized and Multi-Granularity Federated Learning, IEEE INFOCOM, 2024 [Paper]
• Fair Federated Learning with Biased Vision-Language Models, ACL 2024 [Paper]
• Lightweight Unsupervised Federated Learning with Pretrained Vision Language Model, arXiv, 2024 [Paper]

📖 Citation

If you find this survey valuable, please cite:

@article{https://doi.org/10.1002/widm.70036,
author = {Shinde, Gaurav and Ravi, Anuradha and Dey, Emon and Sakib, Shadman and Rampure, Milind and Roy, Nirmalya},
title = {A Survey on Efficient Vision-Language Models},
journal = {WIREs Data Mining and Knowledge Discovery},
volume = {15},
number = {3},
pages = {e70036},
keywords = {edge devices, efficient vision language models, multimodal models},
doi = {https://doi.org/10.1002/widm.70036},
url = {https://wires.onlinelibrary.wiley.com/doi/abs/10.1002/widm.70036},
eprint = {https://wires.onlinelibrary.wiley.com/doi/pdf/10.1002/widm.70036},
note = {e70036 DMKD-00898.R1},
year = {2025}
}