Awesome Machine Unlearning

July 5, 2026 Β· View on GitHub

Awesome arXiv Static Badge Website GitHub stars visitors Contrib

A collection of academic articles, published methodology, and datasets on the subject of machine unlearning.

A sortable version is available here: https://awesome-machine-unlearning.github.io/

Please read and cite our paper: ACM

Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. 2025. A Survey of Machine Unlearning. ACM Trans. Intell. Syst. Technol. Just Accepted (July 2025). https://doi.org/10.1145/3749987

πŸ”– News!!!

πŸ“Œ We are actively tracking the latest research and welcome contributions to our repository and survey paper. If your studies are relevant, please feel free to create an issue or a pull request.

πŸ“° 2025-07-22: Our work has been published in ACM Transactions on Intelligent Systems and Technology. Thanks a lot for your continuous supports.

πŸ“° 2025-02-01: Our work has been cited in the International AI Safety Report 2025 in which machine unlearning is a pioneering paradigm to remove sensitive information or harmful data from trained AI models.

Citation

@article{10.1145/3749987,
  author = {Nguyen, Thanh Tam and Huynh, Thanh Trung and Ren, Zhao and Nguyen, Phi Le and Liew, Alan Wee-Chung and Yin, Hongzhi and Nguyen, Quoc Viet Hung},
  title = {A Survey of Machine Unlearning},
  year = {2025},
  volume = {16},
  number = {5},
  pages = {1--46},
  journal = {ACM Trans. Intell. Syst. Technol.},
}

@article{nguyen2022survey,
  title={A Survey of Machine Unlearning},
  author={Nguyen, Thanh Tam and Huynh, Thanh Trung and Ren, Zhao and Nguyen, Phi Le and Liew, Alan Wee-Chung and Yin, Hongzhi and Nguyen, Quoc Viet Hung},
  journal={arXiv preprint arXiv:2209.02299},
  year={2022}
}

A Framework of Machine Unlearning

timeline Frameworks provide standardized environments, benchmarks, and reproducible research pipelines for studying and evaluating machine unlearning. These works typically focus on methodology, reproducibility, and infrastructure, rather than proposing new unlearning algorithms.

Existing Surveys

Paper TitleVenueYear
Machine Unlearning in Generative AI: A SurveyarXiv2024
SoK: Challenges and Opportunities in Federated UnlearningarXiv2024
Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation MetricsarXiv2024
Digital Forgetting in Large Language Models: A Survey of Unlearning MethodsarXiv2024
Rethinking Machine Unlearning for Large Language ModelsarXiv2024
Threats, Attacks, and Defenses in Machine Unlearning: A SurveyarXiv2024
Machine Unlearning: Solutions and ChallengesTETCI2024
A Survey on Federated Unlearning: Challenges, Methods, and Future DirectionsarXiv2023
A Survey of Federated Unlearning: A Taxonomy, Challenges and Future DirectionsarXiv2023
Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and TaxonomyarXiv2023
Machine Unlearning: A SurveyCSUR2023
An Introduction to Machine UnlearningarXiv2022
Machine Unlearning: Its Need and Implementation StrategiesIC32021
Making machine learning forgetAnnual Privacy Forum2019
β€œAmnesia” - A Selection of Machine Learning Models That Can Forget User Data Very FastCIDR2019
Humans forget, machines remember: Artificial intelligence and the Right to Be ForgottenComputer Law & Security Review2018
Algorithms that remember: model inversion attacks and data protection lawPhilosophical Transactions of the Royal Society A2018

Model-Agnostic Approaches

Model-Agnostic Model-agnostic machine unlearning methodologies include unlearning processes or frameworks that are applicable for different models. In some cases, they provide theoretical guarantees for only a class of models (e.g. linear models). But we still consider them model-agnostic as their core ideas are applicable to complex models (e.g. deep neural networks) with practical results.

Paper TitleYearAuthorVenueModelCodeType
Forgetting Similar Samples: Can Machine Unlearning Do it Better?2026Xu et al.arXiv--Evaluation
Forgetting to Witness: Efficient Federated Unlearning and Its Visible Evaluation2026Wang et al.arXivSkyeye-Federated Unlearning
Machine Unlearning of Personally Identifiable Information in Large Language Models2025Parii et al.NLLPPerMUtokCodeKnowledge Adaptation
The forget-set identification problem2025D'Angelo et al.Machine Learning--Dataset Selection
ESC: Erasing Space Concept for Knowledge Deletion2025Lee et al.CVPRESC[Code]Knowledge Deletion
Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks2025Zhou et al.CVPRDELETE[Code]Remain-data Free
The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Models2025Xiao et al.NeurIPS-RegMLUn-pruning-Weight Pruning
LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty2025Spartalis et al.CVPRLoTUS[Code]Uncertainty-based
FG-OrIU: Towards Better Forgetting via Feature-Gradient Orthogonality for Incremental Unlearning2025Feng et al.ICCVFG-OrIU-Incremental Unlearning
Train Once, Forget Precisely: Anchored Optimization for Efficient Post-Hoc Unlearning2025Sanga et al.ICMLFAMR-Constrained Optimization
Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Unlearning Completeness2025Wang et al.USENIX SecurityIAM[Code]Evaluation
PRUNE: A Patching Based Repair Framework for Certifiable Unlearning of Neural Networks2025Li et al.arXivPRUNE-Certifiable Unlearning
Evaluating the Defense Potential of Machine Unlearning against Membership Inference Attacks2025Tsiolakis et al.arXiv--Evaluation
Potion: Towards Poison Unlearning2024Schoepf et al.DMLRPOTION[Code]Poison Unlearning
Towards Privacy-Guaranteed Label Unlearning in Vertical Federated Learning: Few-Shot Forgetting without Disclosure2024Gu et al.arXiv--Federated Unlearning
Releasing Malevolence from Benevolence: The Menace of Benign Data on Machine Unlearning2024Ma et al.arXiv--Unlearning Usability Attack
Communication Efficient and Provable Federated Unlearning2024Tao et al.VLDBFATS[Code]Federated Unlearning
Sequential Informed Federated Unlearning: Efficient and Provable Client Unlearning in Federated Optimization2024Fraboni et al.AISTATSSIFU[Code]Differential Privacy, Federated Unlearning
Fast-FedUL: A Training-Free Federated Unlearning with Provable Skew Resilience2024Huynh et al.ECML-PKDDFast-FedUL[Code]Federated Unlearning
FRAMU: Attention-based Machine Unlearning using Federated Reinforcement Learning2024Shaik et al.TKDEFRAMU-Federated Learning, Reinforcement Learning
Layer Attack Unlearning: Fast and Accurate Machine Unlearning via Layer Level Attack and Knowledge Distillation2024Kim et al.AAAILAU-Knowledge Adapation
Federated Unlearning: a Perspective of Stability and Fairness2024Shao et al.arXivStability, Fairness, Verification-Federated Unlearning
On the Trade-Off between Actionable Explanations and the Right to be Forgotten2024Pawelczyk et al.arXiv--
Post-Training Attribute Unlearning in Recommender Systems2024Chen et al.arXiv--PoT-AU
CovarNav: Machine Unlearning via Model Inversion and Covariance Navigation2024Abbasi et al.arXivCovarNav-
Partially Blinded Unlearning: Class Unlearning for Deep Networks a Bayesian Perspective2024Panda et al.arXivPBU-
Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning2024Liang et al.arXivUBT-
βˆ‡Ο„: Gradient-based and Task-Agnostic machine Unlearning2024Trippa et al.arXiv--
Towards Independence Criterion in Machine Unlearning of Features and Labels2024Han et al.arXiv--
Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning2024Fan et al.arXiv-[Code]
Corrective Machine Unlearning2024Goel et al.ICLR DMLR-[Code]
Fair Machine Unlearning: Data Removal while Mitigating Disparities2024Oesterling et al.AISTATSfair machine unlearning[Code]
Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models2024Shen et al.arXivLabel-Agnostic Forgetting[Code]
CaMU: Disentangling Causal Effects in Deep Model Unlearning2024Shen et al.arXivCaMU[Code]
SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation2024Fan et al.ICLRSalUn[Code]Weight Saliency
Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening2024Foster et al.AAAISSD[Code]Retraining-free
Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers2024Cha et al.AAAIinstance-wise unlearning[Code]
Parameter-tuning-free data entry error unlearning with adaptive selective synaptic dampening2024Schoepf et al.arXivASSD[Code]
Zero-Shot Machine Unlearning at Scale via Lipschitz Regularization2024Foster et al.arXivJIT[Code]Zero-shot
Is Retain Set All You Need in Machine Unlearning? Restoring Performance of Unlearned Models with Out-Of-Distribution Images2024Bonato et al.arXivSCAR[Code]Knowledge Adaptation
FedCIO: Efficient Exact Federated Unlearning with Clustering, Isolation, and One-shot Aggregation2023Qiu et al.BigDataFedCIO-Federated Unlearning, One-Shot
Towards bridging the gaps between the right to explanation and the right to be forgotten2023Krishna et al.ICML--e
Fast Model DeBias with Machine Unlearning2023Chen et al.NIPSDeBias[Code]
DUCK: Distance-based Unlearning via Centroid Kinematics2023Cotogni et al.arXivDUCK[Code]
Open Knowledge Base Canonicalization with Multi-task Unlearning2023Liu et al.arXivMulCanon-
Unlearning via Sparse Representations2023Shah et al.arXivDKVB-Zero-shot Unlearning
SecureCut: Federated Gradient Boosting Decision Trees with Efficient Machine Unlearning2023Zhang et al.arXivSecureCut-Vertical Federated Learning
Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attacks2023Di et al.NeurIPS--Camouflaged data poisoning attacks
Model Sparsity Can Simplify Machine Unlearning2023Jia et al.NeurIPSl1-sparse[Code]Weight Pruning
Fast Model Debias with Machine Unlearning2023Chen et al.arXiv--
Tight Bounds for Machine Unlearning via Differential Privacy2023Huang et al.arXiv--
Machine Unlearning Methodology base on Stochastic Teacher Network2023Zhang et al.ADMAModel Reconstruction-Knowledge Adaptation
Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening2023Foster et al.arXivSSD[Code]
From Adaptive Query Release to Machine Unlearning2023Ullah et al.arXiv--Exact Unlearning
Towards Adversarial Evaluations for Inexact Machine Unlearning2023Goel et al.arXivEU-k, CF-k[Code]
KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment2023Wang et al.ACLKGA[Code]Knowledge Adaptation
On the Trade-Off between Actionable Explanations and the Right to be Forgotten2023Pawelczyk et al.arXiv--
Towards Unbounded Machine Unlearning2023Kurmanji et al.arXivSCRUB[Code]approximate unlearning
Netflix and Forget: Efficient and Exact Machine Unlearning from Bi-linear Recommendations2023Xu et al.arXivUnlearn-ALS-Exact Unlearning
To Be Forgotten or To Be Fair: Unveiling Fairness Implications of Machine Unlearning Methods2023Zhang et al.arXiv-[Code]
Certified Data Removal in Sum-Product Networks2022Becker and LiebigICKGUNLEARNSPN[Code]Certified Removal Mechanisms
Learning with Recoverable Forgetting2022Ye et al.ECCVLIRF-
Continual Learning and Private Unlearning2022Liu et al.CoLLAsCLPU[Code]
Verifiable and Provably Secure Machine Unlearning2022Eisenhofer et al.arXiv-[Code]Certified Removal Mechanisms
VeriFi: Towards Verifiable Federated Unlearning2022Gao et al.arXivVERIFI-Certified Removal Mechanisms
FedRecover: Recovering from Poisoning Attacks in Federated Learning using Historical Information2022Cao et al.S&PFedRecover-recovery method
Fast Yet Effective Machine Unlearning2022Tarun et al.arXivUNSIR-
Membership Inference via Backdooring2022Hu et al.IJCAIMIB[Code]Membership Inferencing
Forget Unlearning: Towards True Data-Deletion in Machine Learning2022Chourasia et al.ICLR--noisy gradient descent
Zero-Shot Machine Unlearning2022Chundawat et al.arXiv--
Efficient Attribute Unlearning: Towards Selective Removal of Input Attributes from Feature Representations2022Guo et al.arXivattribute unlearning-
Few-Shot Unlearning2022Yoon et al.ICLR--
Federated Unlearning: How to Efficiently Erase a Client in FL?2022Halimi et al.UpML Workshop--federated learning
Machine Unlearning Method Based On Projection Residual2022Cao et al.DSAA--Projection Residual Method
Hard to Forget: Poisoning Attacks on Certified Machine Unlearning2022Marchant et al.AAAI-[Code]Certified Removal Mechanisms
Athena: Probabilistic Verification of Machine Unlearning2022Sommer et al.PoPETsATHENA-
FP2-MIA: A Membership Inference Attack Free of Posterior Probability in Machine Unlearning2022Lu et al.ProvSecFP2-MIA-inference attack
Deletion Inference, Reconstruction, and Compliance in Machine (Un)Learning2022Gao et al.PETS--
Prompt Certified Machine Unlearning with Randomized Gradient Smoothing and Quantization2022Zhang et al.NeurIPSPCMU-Certified Removal Mechanisms
The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retraining2022Liu et al.INFOCOM-[Code]
Backdoor Defense with Machine Unlearning2022Liu et al.INFOCOMBAERASER-Backdoor defense
Markov Chain Monte Carlo-Based Machine Unlearning: Unlearning What Needs to be Forgotten2022Nguyen et al.ASIA CCSMCU-MCMC Unlearning
Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher2022Chundawat et al.arXiv--Knowledge Adaptation
Efficient Two-Stage Model Retraining for Machine Unlearning2022Kim and WooCVPR Workshop--
Learn to Forget: Machine Unlearning Via Neuron Masking2021Ma et al.IEEEForsaken-Mask Gradients
Adaptive Machine Unlearning2021Gupta et al.NeurIPS-[Code]Differential Privacy
Descent-to-Delete: Gradient-Based Methods for Machine Unlearning2021Neel et al.ALT--Certified Removal Mechanisms
Remember What You Want to Forget: Algorithms for Machine Unlearning2021Sekhari et al.NeurIPS--
FedEraser: Enabling Efficient Client-Level Data Removal from Federated Learning Models2021Liu et al.IWQoSFedEraser[Code]Federated Unlearning
Machine Unlearning via Algorithmic Stability2021Ullah et al.COLTTV-Certified Removal Mechanisms
EMA: Auditing Data Removal from Trained Models2021Huang et al.MICCAIEMA[Code]Certified Removal Mechanisms
Knowledge-Adaptation Priors2021Khan and SwaroopNeurIPSK-prior[Code]Knowledge Adaptation
PrIU: A Provenance-Based Approach for Incrementally Updating Regression Models2020Wu et al.NeurIPSPrIU-Knowledge Adaptation
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks2020Golatkar et al.CVPR--Certified Removal Mechanisms
Learn to Forget: User-Level Memorization Elimination in Federated Learning2020Liu et al.arXivForsaken-
Certified Data Removal from Machine Learning Models2020Guo et al.ICML--Certified Removal Mechanisms
Class Clown: Data Redaction in Machine Unlearning at Enterprise Scale2020Felps et al.arXiv--Decremental Learning
A Novel Online Incremental and Decremental Learning Algorithm Based on Variable Support Vector Machine2019Chen et al.Cluster Computing--Decremental Learning
Making AI Forget You: Data Deletion in Machine Learning2019Ginart et al.NeurIPS--Decremental Learning
Lifelong Anomaly Detection Through Unlearning2019Du et al.CCS--
Learning Not to Learn: Training Deep Neural Networks With Biased Data2019Kim et al.CVPR--
Efficient Repair of Polluted Machine Learning Systems via Causal Unlearning2018Cao et al.ASIACCSKARMA[Code]
Understanding Black-box Predictions via Influence Functions2017Koh et al.ICML-[Code]Certified Removal Mechanisms
Towards Making Systems Forget with Machine Unlearning2015Cao and YangS&P-
Towards Making Systems Forget with Machine Unlearning2015Cao et al.S&P--Statistical Query Learning
Incremental and decremental training for linear classification2014Tsai et al.KDD-[Code]Decremental Learning
Multiple Incremental Decremental Learning of Support Vector Machines2009Karasuyama et al.NIPS--Decremental Learning
Incremental and Decremental Learning for Linear Support Vector Machines2007Romero et al.ICANN--Decremental Learning
Decremental Learning Algorithms for Nonlinear Langrangian and Least Squares Support Vector Machines2007Duan et al.OSB--Decremental Learning
Multicategory Incremental Proximal Support Vector Classifiers2003Tveit et al.KES--Decremental Learning
Incremental and Decremental Proximal Support Vector Classification using Decay Coefficients2003Tveit et al.DaWak--Decremental Learning
Incremental and Decremental Support Vector Machine Learning2000Cauwenberg et al.NeurIPS--Decremental Learning

Model-Intrinsic Approaches

Model-Intrinsic The model-intrinsic approaches include unlearning methods designed for a specific type of models. Although they are model-intrinsic, their applications are not necessarily narrow, as many ML models can share the same type.

Paper TitleYearAuthorVenueModelCodeType
Beyond Superficial Forgetting: Thorough Unlearning Through Knowledge Density Estimation and Block Re-Insertion2026Guo et al.AAAILLMCodeKnowledge Adaptation
Roots Beneath the Cut: Uncovering the Risk of Concept Revival in Pruning-Based Unlearning for Diffusion Models2026Zhang et al.CVPR--Pruning-based Unlearning
Machine Unlearning using Forgetting Neural Networks2026Hatua et al.ICAARTFNN-Architecture-based
Secure Forgetting: A Framework for Privacy-Driven Unlearning in Large Language Model (LLM)-Based Agents2026Ye et al.arXiv--LLM Agents
Beyond Forgetting: Machine Unlearning Elicits Controllable Side Behaviors and Capabilities2026Dang et al.arXiv--LLM
SineProject: Machine Unlearning for Stable Vision Language Alignment2025Garg et al.arXivMLLM-Multimodal Unlearning
Stable Forgetting: Bounded Parameter-Efficient Unlearning in LLMs2025Garg et al.arXivLLM-Parameter-Efficient Unlearning
Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models2025Mekala et al.COLINGAltPO[Code]LLM
Sculpting Memory: Multi-Concept Forgetting in Diffusion Models via Dynamic Mask and Concept-Aware Optimization2025Li et al.ICCVDiffusion Models[Code]Concept Unlearning
Efficient Knowledge Graph Unlearning with Zeroth-order Information2025Xiao et al.CIKMKGE[Code]Knowledge Graphs
Knowledge Graph Unlearning with Schema2025Xiao et al.COLINGKGE[Code]Knowledge Graphs
Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech2025Kim et al.ICMLTTS[Code]Audio Generation
No Encore: Unlearning as Opt-out in Music Generation2025Kim et al.NeurIPS-AI4MusicMusic Gen.-Generative Models
Pre-Forgettable Models: Prompt Learning as a Native Mechanism for Unlearning2025Hendrix et al.ACM-MMPFM[Code]Prompt Learning
Intrinsic Test of Unlearning Using Parametric Knowledge Traces2025Hong et al.EMNLPConceptVectors[Code]LLM, Evaluation
Preserving Privacy Without Compromising Accuracy: Machine Unlearning for Handwritten Text Recognition2025Kang et al.arXivWIC-Neural Pruning
Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning2025Zeng et al.arXivSMFA-Multimodal Unlearning
Improving Fisher Information Estimation and Efficiency for LoRA-based LLM Unlearning2025Kim et al.COLMVILA-LLM, Parameter-Efficient
When Forgetting Builds Reliability: LLM Unlearning for Reliable Hardware Code Generation2025Liang et al.arXiv--LLM
Forgetting-MarI: LLM Unlearning via Marginal Information Regularization2025Xu et al.arXivForgetting-MarI-LLM
An Adversarial Perspective on Machine Unlearning for AI Safety2024Łucki et al.TMLR--LLM, Adversarial
Machine Unlearning for Image-to-Image Generative Models2024Li et al.ICLR-[Code]Generative Models
Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models2024Zhang et al.arXivAdvUnlearn[Code]Generative Models, Diffusion Models
Towards Efficient and Effective Unlearning of Large Language Models for Recommendation2024Wang et al.arXivE2URec[Code]LLM
Dissecting Language Models: Machine Unlearning via Selective Pruning2024Pochinkov and SchootsarXivselective pruning[Code]LLM
Decoupling the Class Label and the Target Concept in Machine Unlearning2024Zhu et al.arXivTARF-LLM
Unlearning with Control: Assessing Real-world Utility for Large Language Model Unlearning2024Wang et al.arXivBenchmark-LLM
Decentralized Federated Unlearning on Blockchain2024Liu et al.arXivBoclkFUL-Directed Acyclic Graph
Unlink to Unlearn: Simplifying Edge Unlearning in GNNs2024Tan et al.WWWUtU[Code]GNN-based models
Towards Effective and General Graph Unlearning via Mutual Evolution2024Li et al.AAAIMEGU-GNN-based models
Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operation2024Hu et al.AAAIExt-Sub[Code]LLM
Feature Unlearning for Pre-trained GANs and VAEs2024Moon et al.AAAI--GAN, VAEs
Multi-Modal Recommendation Unlearning2024Sinha et al.arXivMMRecUN-recommendation system
Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models2023Kassem et al.EMNLPDeMem-LLM
Making Users Indistinguishable: Attribute-wise Unlearning in Recommender Systems2023Li et al.MMPoT-AU-Recommendation Systems
FAST: Feature Aware Similarity Thresholding for Weak Unlearning in Black-Box Generative Models2023Panda and AParXivFAST[Code]GAN
Unlearn What You Want to Forget: Efficient Unlearning for LLMs2023Chen & YangEMNLPEUL[Code]LLM
DeltaBoost: Gradient Boosting Decision Trees with Efficient Machine Unlearning2023Wu et al.SIGMODDeltaBoost[Code]Tree-based Models
Federated Unlearning for On-Device Recommendation2023Yuan et al.WSDMFRU-Federated Unlearning, Recommendation Systems
Fast-NTK: Parameter-Efficient Unlearning for Large-Scale Models2023Li et al.arXivFast-NTK-CNN model
Certified Minimax Unlearning with Generalization Rates and Deletion Capacity2023Liu et al.NeurIPS--Minimax model
FairSISA: Ensemble Post-Processing to Improve Fairness of Unlearning in LLMs2023Kadhe et al.NeurIPS-SoLaRFairSISA-LLM
Multimodal Machine Unlearning2023Cheng and AmiriarXivMMUL-Multimodal Models
Adapt then Unlearn: Exploiting Parameter Space Semantics for Unlearning in Generative Adversarial Networks2023Tiwary et al.arXivAdapt-then-Unlearn-GAN
MUter: Machine Unlearning on Adversarially Trained Models2023Liu et alICCVMUter-Adversarial Training Models
Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearning2023Zhu et al.WWWFedLU[Code]GNN-based Models
One-Shot Machine Unlearning with Mnemonic Code2023YamashitaarXivOne-Shot MU-
Inductive Graph Unlearning2023Wang et al.USENIXGUIDE[Code]GNN-based Models
ERM-KTP: Knowledge-level Machine Unlearning via Knowledge Transfer2023Lin et al.CVPRERM-KTP[Code]DNN, Knowledge Adaptation
GNNDelete: A General Strategy for Unlearning in Graph Neural Networks2023Cheng et al.ICLRGNNDELETE[Code]GNN-based Models
Unfolded Self-Reconstruction LSH: Towards Machine Unlearning in Approximate Nearest Neighbour Search2023Tan et al.arXivUSR-LSH[Code]
Efficiently Forgetting What You Have Learned in Graph Representation Learning via Projection2023Cong and MahdaviAISTATSPROJECTOR[Code]GNN-based Models
Unlearning Graph Classifiers with Limited Data Resources2023Pan et al.WWWGST-Unlearn[Code]GNN-based Models
Unrolling SGD: Understanding Factors Influencing Machine Unlearning2022Thudi et al.EuroS&P-[Code]SGD
Graph Unlearning2022Chen et al.CCSGraphEraser[Code]Graph Neural Networks
Certified Graph Unlearning2022Chien et al.GLFrontiers Workshop-[Code]Graph Neural Networks
Skin Deep Unlearning: Artefact and Instrument Debiasing in the Context of Melanoma Classification2022Bevan and Atapour-AbarghoueiICML-[Code]CNN Models
Near-Optimal Task Selection for Meta-Learning with Mutual Information and Online Variational Bayesian Unlearning2022Chen et al.AISTATS--Bayesian Models
Unlearning Protected User Attributes in Recommendations with Adversarial Training2022Ganhor et al.SIGIRADV-MULTVAE[Code]Autoencoder-based Model
Recommendation Unlearning2022Chen et al.TheWebConfRecEraser[Code]Attention-based Model
Knowledge Neurons in Pretrained Transformers2022Dai et al.ACL-[Code]Transformers
Memory-Based Model Editing at Scale2022Mitchell et al.MLRSERAC[Code]DNN-based Models
Forgetting Fast in Recommender Systems2022Liu et al.arXivAltEraser-recommendation system
Deep Regression Unlearning2022Tarun et al.arXivBlindspot-Regression Model
Quark: Controllable Text Generation with Reinforced Unlearning2022Lu et al.NeurIPSQuark[Code]Language Models
Forget-SVGD: Particle-Based Bayesian Federated Unlearning2022Gong et al.DSL WorkshopForget-SVGD-Bayesian Models
Machine Unlearning of Federated Clusters2022Pan et al.arXivSCMA-Federated clustering
Machine Unlearning for Image Retrieval: A Generative Scrubbing Approach2022Zhang et al.MM--DNN-based Models
Machine Unlearning: Linear Filtration for Logit-based Classifiers2022Baumhauer et al.Machine Learningnormalizing filtration-Softmax classifiers
Deep Unlearning via Randomized Conditionally Independent Hessians2022Mehta et al.CVPRL-CODEC[Code]DNN-based Models
Challenges and Pitfalls of Bayesian Unlearning2022Rawat et al.UPML Workshop--Bayesian Models
Federated Unlearning via Class-Discriminative Pruning2022Wang et al.WWW--CNN-Based
Active forgetting via influence estimation for neural networks2022Meng et al.Int. J. Intel. SystemsSCRUBBER-Neural Network
Variational Bayesian unlearning2022Nguyen et al.NeurIPSVI-Bayesian Models
Revisiting Machine Learning Training Process for Enhanced Data Privacy2021Goyal et al.IC3--DNN-based Models
Knowledge Removal in Sampling-based Bayesian Inference2021Fu et al.ICLR-[Code]Bayesian Models
Mixed-Privacy Forgetting in Deep Networks2021Golatkar et al.CVPR--DNN-based Models
HedgeCut: Maintaining Randomised Trees for Low-Latency Machine Unlearning2021Schelter et al.SIGMODHedgeCut[Code]Tree-based Models
A Unified PAC-Bayesian Framework for Machine Unlearning via Information Risk Minimization2021Jose et al.MLSPPAC-Bayesian-Bayesian Models
DeepObliviate: A Powerful Charm for Erasing Data Residual Memory in Deep Neural Networks2021He et al.arXivDEEPOBLIVIATE-DNN-based Models
Approximate Data Deletion from Machine Learning Models: Algorithms and Evaluations2021Izzo et al.AISTATSPRU[Code]Linear/Logistics models
Bayesian Inference Forgetting2021Fu et al.arXivBIF[Code]Bayesian Models
Approximate Data Deletion from Machine Learning Models2021Izzo et al.AISTATSPRU[Code]Linear Models
Online Forgetting Process for Linear Regression Models2021Li et al.AISTATSFIFD-OLS-Linear Models
RevFRF: Enabling Cross-domain Random Forest Training with Revocable Federated Learning2021Liu et al.IEEERevFRF-Random Forrests
Coded Machine Unlearning2021Aldaghri et al.IEEE Access--Deep Learning Models
Machine Unlearning for Random Forests2021Brophy and LowdICMLDaRE RF-Random Forrest
Bayesian Variational Federated Learning and Unlearning in Decentralized Networks2021Gong et al.SPAWC--Bayesian Models
Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations2020Golatkar et al.ECCV--DNN-based Models
Influence Functions in Deep Learning Are Fragile2020Basu et al.arXiv--DNN-based Models
Deep Autoencoding Topic Model With Scalable Hybrid Bayesian Inference2020Zhang et al.IEEEDATM-Bayesian Models
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks2020Golatkar et al.CVPR--DNN-based Models
Uncertainty in Neural Networks: Approximately Bayesian Ensembling2020Pearce et al.AISTATS-[Code]Bayesian Models
Certified Data Removal from Machine Learning Models2020Guo et al.ICML--DNN-based Models
DeltaGrad: Rapid retraining of machine learning models2020Wu et al.ICMLDeltaGrad[Code]DNN-based Models
Making AI Forget You: Data Deletion in Machine Learning2019Ginart et al.NeurIPS--Linear Models
β€œAmnesia” – Towards Machine Learning Models That Can Forget User Data Very Fast2019SchelterAIDB Workshop-[Code]Collaborative Filtering
A Novel Online Incremental and Decremental Learning Algorithm Based on Variable Support Vector Machine2019Chen et al.Cluster Computing--SVM
Neural Text Degeneration With Unlikelihood Training2019Welleck et al.arXivunlikelihood training[Code]DNN-based
Bayesian Neural Networks with Weight Sharing Using Dirichlet Processes2018Roth et al.IEEEDP[Code]Bayesian Models

Data-Driven Approaches

Data-Driven The approaches fallen into this category use data partition, data augmentation and data influence to speed up the retraining process. Methods of attack by data manipulation (e.g. data poisoning) are also included for reference.

Paper TitleYearAuthorVenueModelCodeType
MobText-SISA: Efficient Machine Unlearning for Mobility Logs with Spatio-Temporal and Natural-Language Data2025Yonekura et al.SIGSPATIALMobText-SISA-Data Partitioning
Towards Machine Unlearning Benchmarks: Forgetting the Personal Identities in Facial Recognition Systems2023Choi and NaarXivMUFAC, MUCAC dataset[Code]Introduce two benchmark datasets
SAFE: Machine Unlearning With Shard Graphs2023Dukler et al.ICCVSAFE-Data Partition, Shard Graph
Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attacks2022Di et al.NeurIPS-TSRML-[Code]Data Poisoning
Forget Unlearning: Towards True Data Deletion in Machine Learning2022Chourasia et al.ICLR--Data Influence
ARCANE: An Efficient Architecture for Exact Machine Unlearning2022Yan et al.IJCAIARCANE-Data Partition
PUMA: Performance Unchanged Model Augmentation for Training Data Removal2022Wu et al.AAAIPUMA-Data Influence
Certifiable Unlearning Pipelines for Logistic Regression: An Experimental Study2022Mahadevan and MathioudakisMAKE-[Code]Data Influence
Zero-Shot Machine Unlearning2022Chundawat et al.arXiv--Data Influence
GRAPHEDITOR: An Efficient Graph Representation Learning and Unlearning Approach2022Cong and Mahdavi-GRAPHEDITOR[Code]Data Influence
Fast Model Update for IoT Traffic Anomaly Detection with Machine Unlearning2022Fan et al.IEEE IoT-JViFLa-Data Partition
Learning to Refit for Convex Learning Problems2021Zeng et al.arXivOPTLEARN-Data Influence
Fast Yet Effective Machine Unlearning2021Ayush et al.arXiv--Data Augmentation
Learning with Selective Forgetting2021Shibata et al.IJCAI--Data Augmentation
SSSE: Efficiently Erasing Samples from Trained Machine Learning Models2021Peste et al.NeurIPS-PRIMLSSSE-Data Influence
How Does Data Augmentation Affect Privacy in Machine Learning?2021Yu et al.AAAI-[Code]Data Augmentation
Coded Machine Unlearning2021Aldaghri et al.IEEE--Data Partitioning
Machine Unlearning2021Bourtoule et al.IEEESISA[Code]Data Partitioning
How Does Data Augmentation Affect Privacy in Machine Learning?2021Yu et al.AAAI-[Code]Data Augmentation
Amnesiac Machine Learning2021Graves et al.AAAIAmnesiacML[Code]Data Influence
Unlearnable Examples: Making Personal Data Unexploitable2021Huang et al.ICLR-[Code]Data Augmentation
Descent-to-Delete: Gradient-Based Methods for Machine Unlearning2021Neel et al.ALT--Data Influence
Fawkes: Protecting Privacy against Unauthorized Deep Learning Models2020Shan et al.USENIX Sec. Sym.Fawkes[Code]Data Augmentation
PrIU: A Provenance-Based Approach for Incrementally Updating Regression Models2020Wu et al.SIGMODPrIU/PrIU-opt-Data Influence
DeltaGrad: Rapid retraining of machine learning models2020Wu et al.ICMLDeltaGrad[Code]Data Influence

Datasets

Type: Diffusion

Dataset#ItemsDisk SizeDownstream ApplicationDescription
UnlearnCanvas24K5GB+Style Transfer, etc.UnlearnCanvas: A Stylized Image Dataset to Benchmark Machine Unlearning for Diffusion Models

Type: Image

Dataset#ItemsDisk SizeDownstream ApplicationDescription
MUFAC & MUCAC13K & 30K1GB?Facial RecognitionTowards Machine Unlearning Benchmarks: Forgetting the Personal Identities in Facial Recognition Systems
MNIST70K11MBClassification
CIFAR60K163MBClassification
SVHN600K400MB+Classification
LSUN69M+1TB+Classification
ImageNet14M+166GBClassification

Type: Tabular

Dataset#ItemsDisk SizeDownstream ApplicationDescription
Adult48K+10MBClassification
Breast Cancer569<1MBClassification
Diabetes442<1MBRegression

Type: Text

Dataset#ItemsDisk SizeDownstream ApplicationDescription
Synthetic PII2.25k4.48MBQuestion Answering
TOFU: A Task of Fictitious Unlearning4k6.11MBQuestion Answering
MUSE: Machine Unlearning Six-Way Evaluation for Language Models4k95MBLanguage Modelling
IMDB Review50k66MBSentiment Analysis
Reuters11K+73MBCategorization
Newsgroup20K1GB+Categorization

Type: Sequence

Dataset#ItemsDisk SizeDownstream ApplicationDescription
Epileptic Seizure11K+7MBTimeseries Classification
Activity Recognition10K+26MBTimeseries Classification
Botnet72M3GB+Clustering

Type: Graph

Dataset#ItemsDisk SizeDownstream ApplicationDescription
OGB100M+59MBClassification
Cora2K+4.5MBClassification
MovieLens1B+3GB+Recommender Systems

Evaluation Metrics

MetricsFormula/DescriptionUsage
AccuracyAccuracy on unlearned model on forget set and retrain setEvaluating the predictive performance of unlearned model
CompletenessThe overlapping (e.g. Jaccard distance) of output space between the retrained and the unlearned modelEvaluating the indistinguishability between model outputs
Unlearn timeThe amount of time of unlearning requestEvaluating the unlearning efficiency
Relearn TimeThe epochs number required for the unlearned model to reach the accuracy of source modelEvaluating the unlearning efficiency (relearn with some data sample)
Layer-wise DistanceThe weight difference between original model and retrain modelEvaluate the indistinguishability between model parameters
Activation DistanceAn average of the L2-distance between the unlearned model and retrained model’s predicted probabilities on the forget setEvaluating the indistinguishability between model outputs
JS-DivergenceJensen-Shannon divergence between the predictions of the unlearned and retrained modelEvaluating the indistinguishability between model outputs
Membership Inference AttackRecall (#detected items / #forget items)Verify the influence of forget data on the unlearned model
ZRF scoreZRF=1βˆ’1nfβˆ‘i=0nfJS(M(xi),Td(xi))\mathcal{ZRF} = 1 - \frac{1}{nf}\sum\limits_{i=0}^{n_f} \mathcal{JS}(M(x_i), T_d(x_i))The unlearned model should not intentionally give wrong output \(\mathcal{ZRF} = 0\) or random output \(\mathcal{ZRF} = 1\) on the forget item
Anamnesis Index (AIN)AIN=rt(Mu,Morig,Ξ±)rt(Ms,Morig,Ξ±)AIN = \frac{r_t (M_u, M_{orig}, \alpha)}{r_t (M_s, M_{orig}, \alpha)}Zero-shot machine unlearning
Epistemic Uncertaintyif \mboxi(w;D)>0\mbox{i(w;D) > 0}, then \mboxefficacy(w;D)=1i(w;D)\mbox{efficacy}(w;D) = \frac{1}{i(w; D)};
otherwise \mboxefficacy(w;D)=∞\mbox{efficacy}(w;D) = \infty		How much information the model exposes
Model Inversion AttackVisualizationQualitative verifications and evaluations

Benchmarks

How to Make Reproducible Research in Machine Unlearning with ERASURE published at IJCAI'25

ERASURE: A Modular and Extensible Framework for Machine Unlearning published at CIKM'25

Unlearn PII - Benchmark designed to evaluate the effectiveness of PII unlearning methods, addressing limitations like evaluation implicit knowledge unlearning and assesing all tokens equally

Open Unlearning - An easily extensible framework unifying LLM unlearning evaluation benchmarks

Vision Unlearning - A framework for unlearning algorithms, datasets, metrics, and evaluation methodologies commonly used in Machine Unlearning for vision-related tasks, such as image classification and image generation

Unlearning Comparator - A visual analytics system for comparative evaluation of machine unlearning methods [Paper] [Demo]

UDS (Unlearning Depth Score) - A white-box metric that measures unlearning depth via two-stage activation patching on internal hidden states, evaluated on TOFU forget10 [Paper] [Project Page]

Disclaimer

Feel free to contact us if you have any queries or exciting news on machine unlearning. In addition, we welcome all researchers to contribute to this repository and further contribute to the knowledge of machine unlearning fields.

If you have some other related references, please feel free to create a Github issue with the paper information. We will glady update the repos according to your suggestions. (You can also create pull requests, but it might take some time for us to do the merge)

Star History Chart

Backup Statistics

Hits HitCount

Hits

Visitor Count

Visitors