✨ Awesome PEFT A2Z ✨

PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models

📝 Paper

If you just want a quick lookup:

PEFT A2Z: Parameter‑Efficient Fine‑Tuning Survey for Large Language and Vision Models

💡 Abstract

Large models such as Large Language Models (LLMs) and Vision Language Models (VLMs) have transformed artificial intelligence, powering applications in natural language processing, computer vision, and multimodal learning. However, fully fine-tuning these models remains expensive, requiring extensive computational resources, memory, and task-specific data. Parameter-Efficient Fine-Tuning (PEFT) has emerged as a promising solution that allows adapting large models to downstream tasks by updating only a small portion of parameters. This survey presents a comprehensive overview of PEFT techniques, focusing on their motivations, design principles, and effectiveness. We begin by analyzing the resource and accessibility challenges posed by traditional fine-tuning and highlight key issues, such as overfitting, catastrophic forgetting, and parameter inefficiency. We then introduce a structured taxonomy of PEFT methods—grouped into additive, selective, reparameterized, hybrid, and unified frameworks—and systematically compare their mechanisms and trade-offs. Beyond taxonomy, we explore the impact of PEFT across diverse domains, including language, vision, and generative modeling, showing how these techniques offer strong performance with lower resource costs. We also discuss important open challenges in scalability, interpretability, and robustness, and suggest future directions such as federated learning, domain adaptation, and theoretical grounding. Our goal is to provide a unified understanding of PEFT and its growing role in enabling practical, efficient, and sustainable use of large models.

⭐ Citation

If you find our work useful, please cite it as:

@article{prottasha2025peft,
  title={PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models},
  author={Prottasha, Nusrat Jahan and Chowdhury, Upama Roy and Mohanto, Shetu and Nuzhat, Tasfia and Sami, Abdullah As and Ali, Md Shamol and Sobuj, Md Shohanur Islam and Raman, Hafijur and Kowsher, Md and Garibay, Ozlem Ozmen},
  journal={arXiv preprint arXiv:2504.14117},
  year={2025}
}

🚀 Serial Adapters

• Parameter-Efficient Transfer Learning for NLP [Paper]

• AdapterHub: A Framework for Adapting Transformers [Paper]

• MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer [Paper]

• Cross-Lingual Transfer with Target Language-Ready Task Adapters [Paper]

• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning [Paper]

🚀 Parallel Adapters

• UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling [Paper]

• UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory [Paper]

• AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition [Paper]

• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning [Paper]

• PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning [Paper]

• Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference [Paper]

🚀 Hybrid Adapters

• AUTOPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning [Paper]

• CROSS-MODAL ADAPTER: PARAMETER-EFFICIENT TRANSFER LEARNING APPROACH FOR VISION-LANGUAGE MODELS [Paper]

• EFFICIENT REMOTE SENSING WITH HARMONIZED TRANSFER LEARNING AND MODALITY ALIGNMENT [Paper]

• MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval [Paper]

• Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets [Paper]

• Conditional Adapters: Parameter-Efficient Transfer Learning with Fast Inference [Paper]

🚀 Single Task

• VISION TRANSFORMER ADAPTER FOR DENSE PREDICTIONS [Paper]

• Simple, Scalable Adaptation for Neural Machine Translation [Paper]

• K-ADAPTER: Infusing Knowledge into Pre-Trained Models with Adapters [Paper]

🚀 Multi Task

• K-ADAPTER: Infusing Knowledge into Pre-Trained Models with Adapters [Paper]

• AdapterFusion: Non-Destructive Task Composition for Transfer Learning [Paper]

• OrchMoE: Efficient Multi-Adapter Learning with Task-Skill Synergy [Paper]

• Multi-Head Adapter Routing for Cross-Task Generalization [Paper]

• Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks [Paper]

• When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications [Paper]

• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models [Paper]

• AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models [Paper]

🚀 Continuous Prompting

• Prefix-Tuning: Optimizing Continuous Prompts for Generation [Paper]

• PEDRO: Parameter-Efficient Fine-tuning with Prompt DEpenDent Representation MOdification [Paper]

• DEPT: Decomposed Prompt Tuning for Parameter-Efficient Fine-Tuning [Paper]

• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks [Paper]

• Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Models [Paper]

• PTR: Prompt Tuning with Rules for Text Classification [Paper]

• Prefix-Propagation: Parameter-Efficient Tuning for Long Sequences [Paper]

• Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts [Paper]

• OVOR: OnePrompt with Virtual Outlier Regularization for Rehearsal-Free Class-Incremental Learning [Paper]

🚀 Discrete Prompt

• RLPROMPT: Optimizing Discrete Text Prompts with Reinforcement Learning [Paper]

• SPARSEFIT: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations [Paper]

• OVOR: OnePrompt with Virtual Outlier Regularization for Rehearsal-Free Class-Incremental Learning [Paper]

🚀 Domain Specific Adaption (Natural Language Understanding)

• Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptation [Paper]

• InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding [Paper]

• PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization [Paper]

• IDPG: An Instance-Dependent Prompt Generation Method [Paper]

• APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models [Paper]

• SMoP: Towards Efficient and Effective Prompt Tuning with Sparse Mixture-of-Prompts [Paper]

🚀 Task Specific Adaption

• The Power of Scale for Parameter-Efficient Prompt Tuning [Paper]

• SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer [Paper]

• APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference [Paper]

• XPROMPT: Exploring the Extreme of Prompt Tuning [Paper]

• Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts [Paper]

• IDPG: An Instance-Dependent Prompt Generation Method [Paper]

• APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models [Paper]

• SMoP: Towards Efficient and Effective Prompt Tuning with Sparse Mixture-of-Prompts [Paper]

🚀 Scaling Adaption

• Propulsion: Steering LLM with Tiny Fine-Tuning [Paper]

🚀 Selective Tuning Based on Parameter Importance

• Parameter-Efficient Transfer Learning with Diff Pruning [Paper]

• Targeted Efficient Fine-tuning: Optimizing Parameter Updates with Data-Driven Sample Selection [Paper]

• Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models [Paper]

• AdaFish: Fast Low-Rank Parameter-Efficient Fine-Tuning by Using Second-Order Information [Paper]

• Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning [Paper]

🚀 Unstructured Mask

• Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models [Paper]

• Composable Sparse Fine-Tuning for Cross-Lingual Transfer [Paper]

• Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning [Paper]

• Parameter-Efficient Fine-Tuning without Introducing New Latency [Paper]

🚀 Structured Mask

• Efficient Fine-Tuning of BERT Models on the Edge [Paper]

• Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation [Paper]

• X-PEFT: eXtremely Parameter-Efficient Fine-Tuning for Extreme Multi-Profile Scenarios [Paper]

• Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning [Paper]

🚀 Core Low Rank Decomposition

• LoRA: Low-Rank Adaptation of Large Language Models [Paper]

• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers [Paper]

• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning [Paper]

• Parameter-Efficient Model Adaptation for Vision Transformers [Paper]

• Parameter-Efficient Fine-Tuning without Introducing New Latency [Paper]

• DoRA: Weight-Decomposed Low-Rank Adaptation [Paper]

• LLMEmbed: Rethinking Lightweight LLM’s Genuine Function in Text Classification [Paper]

🚀 Adaptive and dynamic rank methods

• DyLoRA: Parameter-Efficient Tuning of Pretrained Models using Dynamic Search-Free Low Rank Adaptation [Paper]

• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning [Paper]

• Sparse Low-Rank Adaptation of Pre-trained Language Models [Paper]

• Increasing Model Capacity for Free: A Simple Strategy for Parameter-Efficient Fine-Tuning [Paper]

• AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning [Paper]

🚀 Enhanced LoRA variants for fine tuning efficiency

• Bayesian Low-Rank Adaptation for Large Language Models [Paper]

• LoRA Dropout as a Sparsity Regularizer for Overfitting Control [Paper]

• PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization [Paper]

• LoRA+: Efficient Low Rank Adaptation of Large Models [Paper]

• Mixture-of-Subspaces in Low-Rank Adaptation [Paper]

• Continual Learning with Low Rank Adaptation [Paper]

• Trans-LoRA: Towards Data-Free Transferable Parameter Efficient Finetuning [Paper]

• RoseLoRA: Row and Column-wise Sparse Low-Rank Adaptation [Paper]

• Low-Rank Few-Shot Adaptation of Vision-Language Models [Paper]

• SVDQUANT: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models [Paper]

• Variational Low-Rank Adaptation Using IVON [Paper]

• PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment [Paper]

• Empower Vision Applications with LoRA LMM [Paper]

• LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition [Paper]

• MOELoRA: An MOE-based Parameter Efficient Fine-Tuning Method for Multi-task Medical Applications [Paper]

• Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning [Paper]

• Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models [Paper]

• MIXTURE OF LORA EXPERTS [Paper]

• MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts [Paper]

🚀 Hybrid Approaches

• Towards a Unified View of Parameter-Efficient Transfer Learning [Paper]

• UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning [Paper]

• Parameter-Efficient Fine-Tuning Design Spaces [Paper]

• Neural Prompt Search [Paper]

• AUTOPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning [Paper]

• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models [Paper]

• RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation [Paper]

• Sparsity- and Hybridity-Inspired Visual Parameter-Efficient Fine-Tuning for Medical Diagnosis [Paper]

• HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks [Paper]

• Hydra: Multi-head Low-rank Adaptation for Parameter Efficient Fine-tuning [Paper]

🚀 MoE Based

• When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications [Paper]

• MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts [Paper]

• Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning [Paper]

• Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models [Paper]

• Mixture-of-Subspaces in Low-Rank Adaptation [Paper]

📊 Summary Tables

We present a comprehensive summary of PEFT methods applied in NLP tasks across various LLMs in Tables 8, 9, 10, 11, and 12 of the paper .

Below is an example (Table 8).

PEFT Methods in NLP Tasks

Models and Papers

Year	Model	Paper Title	Links
15 Oct 2024	Bloomz	Investigating translation for Indic languages with BLOOMZ-3b through prompting and LoRA fine-tuning	Paper
31 July 2024	LLaMA3-8B	The Llama 3 Herd of Models	arXiv GitHub
18 Aug 2024	Recorder	A Syntax-Guided Edit Decoder for Neural Program Repair	Paper
18 June 2024	ChatGLM-6B	ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools	arXiv GitHub
26 June 2024	Tag-LLaMA	Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains	arXiv
31 May 2024	Mamba-2	Mamba: Linear-Time Sequence Modeling with Selective State Spaces	arXiv GitHub
15 May 2024	LLaVA-1.5	Improved Baselines with Visual Instruction Tuning	arXiv GitHub
16 April 2024	GEMMA-2B, Gemma-7B	Gemma: Open Models Based on Gemini Research and Technology	arXiv GitHub
16 Mar 2024	Openchat8B	OpenChat: Advancing Open-source Language Models with Mixed-Quality Data	arXiv GitHub
26 Feb 2024	Airavata-7b	Airavata: Introducing Hindi Instruction-tuned LLM	arXiv
14 Feb 2024	LlaSMol	LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset	arXiv GitHub
4 Jan 2024	TinyLlama	TinyLlama: An Open-Source Small Language Model	arXiv GitHub
2024	DeepSeek-Coder-Base-6.7B	DeepSeek Coder: When the Large Language Model Meets Programming	arXiv GitHub
2024	GPT-J (6B)	GPT-J: 6B Parameter Open Source Transformer Model	GitHub
14 Dec 2024	TigerBot-7B	TigerBot: A Multi-stage Open-Source LLM for Human and Agent Scenarios	arXiv GitHub
30 Nov 2023	CCT5	A Code-Change-Oriented Pre-trained Model	Paper
29 Nov 2023	Falcon	The Falcon Series of Open Language Models	arXiv
21 Nov 2023	ShareGPTv4(7B)	ShareGPT4V: Improving Large Multi-Modal Models with Better Captions	arXiv GitHub
Dec 1 2023	Mamba	Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality	arXiv GitHub
10 Oct 2023	Mistral-7B-Instruct	Mistral 7B	arXiv GitHub
24 Aug 2023	Qwen-VL-Chat(7B)	Qwen-VL: A Vision-Language Foundation Model for Universal Multimodal Understanding and Generation	arXiv GitHub
24 Aug 2023	CodeLlama-7B	Code Llama: Open Foundation Models for Code	arXiv GitHub
18 July 2023	Llama2-7b,3b,13b,70b	Llama 2: Open Foundation and Fine-Tuned Chat Models	arXiv GitHub
27 June 2023	BLOOM-7B,1B	BLOOM: A 176B-Parameter Open-Access Multilingual Language Model	arXiv GitHub
9 June 2023	Vicuna-7b,13b	Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena	arXiv GitHub
15 Mar 2023	GPT-4	GPT-4 Technical Report	arXiv
27 Feb 2023	LLaMA30B	LLaMA: Open and Efficient Foundation Language Models	arXiv GitHub
29 Jan 2023	Text+Chem T5	Unifying Molecular and Textual Representations via Multi-task Language Modelling	arXiv
19 April 2023	Baichuan2-13B	Baichuan 2: Open Large-scale Language Models	arXiv GitHub
16 Nov 2022	Galactica	Galactica: A Large Language Model for Science	arXiv GitHub
5 Oct 2022	PaLM	PaLM: Scaling Language Modeling with Pathways	arXiv
20 Oct 2022	Flan-T5 (Flan-T5-base, Flan-T5-xl)	Scaling Instruction-Finetuned Language Models	arXiv GitHub
Nov 30 2022	GPT-3.5	GPT-3.5 Technical Report	Blog
5 Aug 2022	BlenderBot	BlenderBot 3: A Deployed Conversational Agent That Continually Learns to Responsibly Engage	arXiv
Sep 2022	GIZA++	Embedding-Enhanced GIZA++: Improving Word Alignment Using Embeddings	Paper
25 Mar 2022	CodeGen	CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis	arXiv GitHub
12 April 2022	INCODER-1B,6B	InCoder: A Generative Model for Code Infilling and Synthesis	arXiv GitHub
10 April 2022	RewardRepair	Neural Program Repair with Execution-based Backpropagation	arXiv
2 May 2022	OPT (OPT, OPT-13B, OPT-6.7B, OPT-1.3B)	OPT: Open Pre-trained Transformer Language Models	arXiv GitHub
29 Apr 2021	EFL	Entailment as Few-Shot Learner	arXiv
18 Nov 2021	DeBERTaV3-base	DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing	arXiv GitHub
2 Sep 2021	CodeT5	CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation	arXiv GitHub
Aug 2021	GPT-Neo	GPT-Neo: Implementation of Model Parallel GPT2-like Models	GitHub
7 May 2021	CURE	Code-Aware Neural Machine Translation for Automatic Program Repair	Paper
28 May 2020	GPT-3	Language Models are Few-Shot Learners	arXiv
5 June 2020	DeBERTaLARGE,DeBERTa	DeBERTa: Decoding-enhanced BERT with Disentangled Attention	arXiv GitHub
18 Apr 2020	SimAlign	SimAlign: High Quality Word Alignments Without Parallel Training Data Using Static and Contextualized Embeddings	arXiv GitHub
10 Apr 2020	Longformer	Longformer: The Long-Document Transformer	arXiv GitHub
6 Apr 2020	MobileBERT	MobileBERT: A Compact Task-Agnostic BERT for Resource-Limited Devices	arXiv
27 April 2020	ColBERT	ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT	arXiv GitHub
23 Mar 2020	ELECTRA	ELECTRA: Pre-training Text Encodors as Discriminators Rather Than Generators	arXiv GitHub
4 Mar 2020	jiant	jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models	arXiv GitHub
14 Feb 2020	TwinBERT	TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval	arXiv
29 Oct 2019	BART-Large	BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	arXiv GitHub
23 Oct 2019	T5 (T5-BASE, T5-SMALL, T5-Large)	Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	arXiv GitHub
8 Nov 2019	mBART	How Language-Neutral is Multilingual BERT?	arXiv GitHub
5 Nov 2019	XLM-R (xlm-roberta-base)	Unsupervised Cross-lingual Representation Learning at Scale	arXiv GitHub
1 Nov 2019	DialoGPT	DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation	arXiv GitHub
2 Oct 2019	DistilBERT	DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter	arXiv GitHub
26 Sep 2019	ALBERT	ALBERT: A Lite BERT for Self-supervised Learning of Language Representations	arXiv GitHub
26 July 2019	RoBERTa (RoBERTa-Large, RoBERTa-Base)	RoBERTa: A Robustly Optimized BERT Pretraining Approach	arXiv GitHub
19 June 2019	XLNet	XLNet: Generalized Autoregressive Pretraining for Language Understanding	arXiv GitHub
Feb 2019	GPT2 (GPT2, GPT2-M, GPT2-L)	Language Models are Unsupervised Multitask Learners	Paper GitHub
11 Oct 2018	BERT-Base,Large	BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	arXiv GitHub
12 June 2017	Transformer	Attention Is All You Need	arXiv GitHub
July 2002	BLEU	BLEU: A Method for Automatic Evaluation of Machine Translation	Paper

📊 Summary Tables

We present a comprehensive summary of PEFT methods applied in vision tasks across various LLMs in Tables 13 and 14 .

Below is an example (Table 13).

PEFT Methods in Vision Models

Models and Papers

Year	Model	Paper Title	Links
7 Mar 2025	FaceT-B, PLFace	Efficient Fine-tuning Strategies for Enhancing Face Recognition Performance in Challenging Scenarios	Paper
5 Apr 2023	SAM	Segment Anything	arXiv GitHub
24 Mar 2023	Point-PEFT	Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Models	Paper
16 Mar 2023	ESM2	Evolutionary-scale prediction of atomic protein structure with a language model	Paper GitHub
20 Dec 2022	DiT	Scalable Diffusion Models with Transformers	arXiv GitHub
21 Feb 2022	PointM2AE	Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training	Paper
23 Mar 2022	VideoMAE	VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training	arXiv GitHub
16 Mar 2022	CodeReviewer	Automating Code Review Activities by Large Language Models	arXiv
13 Mar 2022	Point-MAE	Point-MAE: Masked Autoencoders for 3D Point Cloud Self-supervised Learning	arXiv GitHub
11 Jan 2022	ConvNeXt	A ConvNet for the 2020s	arXiv GitHub
29 Nov 2021	Point-BERT	Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling	arXiv GitHub
11 Nov 2021	MAE	Masked Autoencoders Are Scalable Vision Learners	arXiv GitHub
24 Jun 2021	Swin Video Transformer	Video Swin Transformer	arXiv GitHub
31 May 2021	Segformer	SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers	arXiv GitHub
18 Jul 2021	AS-MLP	AS-MLP: An Axial Shifted MLP Architecture for Vision	arXiv
25 Mar 2021	Swin Transformer (Swin-L, Swin-B)	Swin Transformer: Hierarchical Vision Transformer using Shifted Windows	arXiv GitHub
26 Feb 2021	CLIP	Learning Transferable Visual Models From Natural Language Supervision	arXiv GitHub
24 Feb 2021	PVT	Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions	arXiv GitHub
11 Feb 2021	NFNets	High-Performance Large-Scale Image Recognition Without Normalization	arXiv GitHub
31 Dec 2020	SETR	Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers	arXiv GitHub
23 Dec 2020	DeiT	Training data-efficient image transformers & distillation through attention	arXiv GitHub
22 Oct 2020	ViT (ViT-B/16, ViT-S)	An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale	arXiv GitHub
7 Mar 2020	RegNetX	Designing Network Design Spaces	arXiv GitHub
13 Feb 2020	SimCLR	A Simple Framework for Contrastive Learning of Visual Representations	arXiv GitHub
30 Jun 2018	SELDnet	Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks	arXiv
17 Mar 2017	ProtoNet	Prototypical Networks for Few-shot Learning	arXiv GitHub
23 May 2016	WideResNet	Wide Residual Networks	arXiv GitHub
10 Dec 2015	ResNet (ResNet-50, ResNet-101)	Deep Residual Learning for Image Recognition	arXiv GitHub
4 Sep 2014	VGG (VGG16, VGG19)	Very Deep Convolutional Networks for Large-Scale Image Recognition	arXiv GitHub

📊 Summary Tables

We present a comprehensive summary of PEFT methods applied to both NLP and vision tasks across various datasets in Tables 5, 6 and 7 .

Below is an example (Table 5).

🚀 PEFT Methods in NLP Tasks Datasets

Dataset	Task Type	Dataset Size	Dataset Link
20 Newsgroups	Text Classification	20K	Link
CoNLL03	Named Entity Recognition	2302 articles	Link
CoNLL04	Relation Extraction Tasks	1,437 sentences	Link
SST-2	Sentiment Classification	215,154	Link
MRPC	Paraphrase Detection	5,800	Link
ACE2005	Information Extraction (IE)	599 documents	Link
WMT	Machine Translation	N/A	Link
MPQA	Opinion Mining	535 (articles)	Link
Common Crawl	Web Scraping, NLP Pretraining	386 TiB	Link
i2b2 2010 RE	Relation Extraction in Clinical Text	877 documents	Link
Semeval-2010	Semantic Role Labeling, WSD, Relation Extraction	284	Link
COPA	Commonsense Causal Reasoning	1000 questions	Link
IMDb	Sentiment Classification	50k reviews	Link
SBU	Image Captioning	1 million images	Link
WebQ	Question Answering	6,642	Link
OntoNotes	NER, Coreference Resolution	2.9453M	Link
DUT	Salient Object Detection	5,168 images	Link
AG News	Text Classification	1M	Link
Yelp	Sentiment Classification	700,000	Link
VQA v2.0	Visual Question Answering	265,016	Link
TriviaQA	Question Answering	650K	Link
SQuAD	Question Answering	100K	Link
BoolQ	Boolean Question Answering	15,942	Link
YELP-Polarity	Sentiment Analysis (Binary Classification)	560,000	Link
CNN/DailyMail	Abstractive Summarization	313k	Link
Yelp	Sentiment Classification	700,000	Link
WMT 16 en-ro	Machine Translation (English to Romanian and vice versa)	2 million sentences	Link
MAWPS	Math Word Problem Solving	2,373 problems	Link
MIMIC-III MP	Medical Predictive Modeling	60,000 ICU stays	Link
WMT 16 en-ro	Machine Translation (English to Romanian and vice versa)	2 million sentences	Link
E2E	Data to Text Generation	N/A	Link
MNLI	Natural Language Inference	433k	Link
WebNLG	Natural Language Generation	13,211 triples	Link
STS-B	Semantic Textual Similarity	8,628 pairs	Link
HS	Hate Speech Detection	25,000 tweets	Link
WikiSQL	Text-to-SQL Generation	80,654	Link
ARC	Multiple Choice Question Answering	7,787	Link
GLUE	Natural Language Understanding	N/A	Link
OBQA	Commonsense Question Answering	5,957 questions	Link
ARC-Easy	Multiple-choice Question Answering	5,197	Link
ARC-Challenge	Multiple-choice Question Answering	2,590	Link
MultiRC	Multi-Sentence Reading Comprehension	10,000 questions	Link
XSum	Abstractive Summarization	226,711	Link
CoLA	Sentence Acceptability	10,657 sentences	Link
SciTail dataset	Textual Entailment	27,026 examples	Link
SQuAD 2.0	Question Answering	150,000+ question-answer pairs	Link
PIQA	Multiple Choice Question Answering	19K	Link
Winogrande	Commonsense Reasoning	44K	Link
OSCAR	Pre-training Language Models	50K	Link
SuperGLUE	Natural Language Understanding (NLU)	N/A	Link
codeSearchNet	--	2 million	Link
AmazonQA/ProductQA	Question Answering (QA)	923K questions, 3.6M answers	Link
commonsenseQA (CSQA)	Question Answering	12,247 questions	Link
SAMSum	Dialogue Summarization	16,369	Link
Word-in-Context (WiC)	Evaluate Context-Sensitive Meaning	7,466	Link
Hyperpartisan	Binary Classification (Hyperpartisan vs. Non-Hyperpartisan)	754,000 articles	Link
PAWS	Paraphrase Identification	108,463	Link
WoW	Knowledge-driven Dialogue Generation	N/A	Link
CosmosQA	Commonsense Reading Comprehension	35.6K questions	Link
HellaSwag	Commonsense Reasoning / Sentence Completion	N/A	Link
MMLU	Multiple-Choice QA	15,000+ questions across 57 subjects	Link
The Pile	Language Modeling	825GB	Link
WikiTableText	Data-to-Text / QA	100M+ tokens	Link
DART	Data-to-Text	82,191 examples	Link
C4	Language Modeling / Pre-training	806.87 GiB	Link
LibriLight	Self-supervised Speech	60,000 hours	Link
ANLI	Natural Language Inference	162,865 examples	Link
COMMONGEN	Text Generation	79K descriptions over 35K concept sets	Link
GSM8K (Grade School Math 8K)	Mathematical Problem Solving	8,500	Link
MATH	Mathematical Problem Solving	12,500	Link
TruthfulQA	Question Answering, Text Generation	817	Link
Therapeutics Data Commons benchmark	AI-driven drug discovery	66 AI-ready datasets	Link
SVAMP	Math Word Problem Solving	1,000 problems	Link
CUB-200-2011	Fine-grained Image Classification	11,788 images	Link
EL EVATER	Multimodal Entity Linking, Visual Attribute Recognition	10,000 instances (text-image pairs)	Link
TB-1k	Medical Image Classification	1,000K	Link
ro-en	--	614,318 sentences	N/A
de-en	--	4,554,053 sentences	N/A
Ha–En	--	50,000 sentences	N/A
(De-En)	--	N/A	N/A
Fr–De	--	N/A	N/A
Fr–Es	--	N/A	N/A
Flan v2	Instruction tuning for NLP tasks	1,800+	Link
BBH	Question-Answering	6,511	Link
Pix2Struc	Visual Language Understanding	N/A	Link
M2D2	Text	8.5B tokens across 145 domains	Link
Alpaca	Instruction Following	52K	Link
GPT-4 Alpaca	Instruction-following fine-tuning	52K	Link
Dolly	Instruction-Following NLP	15K prompts	Link
Orca	Instruction-Following, Reasoning	1.6 million	Link
GPT-4-Turbo	--	128k	N/A
AI4Bharat Naamapadam	NER on Indian languages	400k sentences	Link
AmericasNLI	Natural Language Inference	N/A	N/A
SIQA	Text	33,410	N/A
TREC	Question Classification	4,500	Link
ScienceQ	Science Question Answering	N/A	N/A
Wikitext2	Language Modeling	2M	Link
Penn Treebank	Language Modeling	1M	N/A
VQA v2.0	Visual Question Answering	265,016	Link
VisDA-C	Domain Adaptation for Image Classification	280k images	Link
ImageNet-Sketch	Image Classification	50,889 images	Link
ImageNet-A	Image Classification	7,500 images	Link
DocVQA	Visual Question Answering (VQA) on documents	50,000 questions, 12,767 images	Link
IWSLT	--	N/A	N/A
FGVC	Fine-grained object classification	10,200 images of aircraft	Link
OPUS-100	Multilingual Machine Translation (text-to-text)	55 million parallel sentences	Link
MS COCO dataset	Object Detection, Instance Segmentation, Panoptic Segmentation, Image Captioning, Keypoint Detection	118K images / 5K images / 41K images	Link
COCO Stuff	Semantic Segmentation, Panoptic Segmentation, Scene Understanding	164,000 images	Link
PASCAL VOC	Object Detection, Semantic Segmentation, Image Classification, Action Recognition, Person Layout	VOC 2012: 11,530 images; Object instances: 27,450; Segmentation: 2,913 images with pixel masks	Link
MICCAI 2015 Multi-Atlas Abdomen Labeling Challenge	Multi-organ Segmentation (13 abdominal organs)	30 CT scans / 20 CT scans	Link
PASCAL Context	Object Detection/Segmentation, Semantic Segmentation, Scene Parsing	10,103 images with 65,937 labeled objects; Classes: 459	Link
Pascal Context-59	Semantic Segmentation, Scene Parsing	10,103 images with dense labels for 59 classes	Link
Pascal Context-459	--	N/A	N/A
ADE20K 150	Semantic Segmentation, Scene Parsing, Instance Segmentation (optional)	25,000 images	Link
ADE20K-847	Semantic Segmentation, Scene Parsing, Instance Segmentation (optional)	25,000 images	Link
CIFAR-10	Image Classification	60,000 images	Link
CIFAR-100	Image Classification	60,000 images	Link
CIFAR-10-LT (Long Tail)	Image Classification	60,000 images	Link
ImageNet1K	Image Classification and Localization	1,431,167 images	Link
ImageNet100	Image Classification	131,689 images	Link
ORBIT	Few-shot learning for teachable object recognition	2,687 videos	Link
ScanObjectNN	3D Object Classification	2,902 object instances	Link
ModelNet40	3D Shape Classification	12,311 3D objects / 2,468 models	Link
RESISC45	Remote Sensing Scene Classification	31,500 RGB images	Link
CLEVR	Visual Question Answering (VQA), Compositional Reasoning	100,000 images / 1 million questions	Link
DepthTrack	RGB-D Single Object Tracking	150 sequences (100,000+ frames)	Link
IconQA	Visual Question Answering	N/A	N/A
ImageNet-R	Image Classification	30,000 images	Link
Pix2Struc	Visual Language Understanding	N/A	N/A
M2D2 (2022)	Multi-task NLP (NER, RE, QA, etc.)	8.5B tokens across 145 domains	Link
Vizwiz	Visual Question Answering	N/A	Link
Flickr30k	Image Captioning	31,783	Link
OKVQA	Visual Question Answering	14,055	Link
OCR-VQA	Optical Character Recognition	N/A	Link
LibriSpeech	Automatic Speech Recognition	1,000 hours	Link
M2D2 (2022 again)	Multi-task NLP (NER, RE, QA, etc.)	8.5B tokens across 145 domains	Link
ImageNet-Sketch	Image Classification	50,889 images	Link
ImageNet-A	Image Classification	7,500 images	Link
ImageNet-C	Image Classification under Corruption	3,750,000 corrupted images	Link
VisDA-C	Domain Adaptation for Image Classification	280K images	Link
DomainNet-126	Domain Adaptation for Image Classification	586,575 images	Link
SBU	Image Captioning	1 million images	Link
OxfordPets	Fine-Grained Classification & Semantic Segmentation	7,390 images	Link
DUT	Salient Object Detection	5,168 images	Link
DTD	Texture Classification	5,640 images	Link
Food101	Food Image Classification	101,000 images	Link
Kinetics-400	Action Recognition, Video Classification	650k video clips	Link
Something-Something-v2	Temporal Action Recognition	220,847 videos	Link
EuroSAT	Land Use/Land Cover Classification	27,000 images	Link
iNaturalist 2018	Fine-Grained Species Classification	611,753 images	Link
IMD20	--	35,000 images	Link
COD10K	Camouflaged Object Detection	10,000 images	Link
CAMO	Camouflaged Object Segmentation	1,250 images	Link
ISTD	Shadow Detection & Removal	N/A	Link
UCF101	Action Recognition, Video Classification	13,320 videos (7.2 GB)	Link
SUN397	Scene Classification	108,753 images	Link
ABIDE	Neuroimaging Classification (ASD vs. Controls)	1,112 subjects	Link
Places-LT	Long-Tailed Scene Classification	62,500 images	Link
CASIA	--	N/A	Link
IMD20	--	N/A	Link
CUHK	--	N/A	Link
CHAMELEON (Confused)	--	N/A	Link
Kinetics-700	--	N/A	Link
DF-20M mini	--	N/A	Link
RIGA+ (Not found)	--	55	Link
SCGM (Not found)	--	N/A	Link
VQA v2.0	Visual Question Answering	265,016	Link

🙌 Contribution

A huge thank you to everyone who’s made this project possible! 🎉