Toward Robust Non-Transferable Learning: A Survey and Benchmark

June 2, 2026 · View on GitHub

arXiv Awesome

Over the past decades, researchers have primarily focused on improving the generalization abilities of models, with limited attention given to regulating such generalization. However, the ability of models to generalize to unintended data (e.g., harmful or unauthorized data) can be exploited by malicious adversaries in unforeseen ways, potentially resulting in violations of model ethics. Non-transferable learning (NTL), a task aimed at reshaping the generalization abilities of deep learning models, was proposed to address these challenges. While numerous methods have been proposed in this field, a comprehensive review of existing progress and a thorough analysis of current limitations remain lacking.

We bridge this gap by 👉 presenting the first comprehensive survey on NTL and 👉 introducing NTLBench, the first benchmark to evaluate NTL performance and robustness within a unified framework.

🧐 We present the comparison of 5 methods (NTL, CUTI-domain, H-NTL, SOPHON, CUPI-domain) on CIFAR & STL with VGG-13, where we evaluate non-transferability performance and post-training robustness against 5 source domain fine-tuning (SourceFT) attacks, 4 target domain fine-tuning (TargetFT) attacks, and 6 source-free domain adaptation (SFDA) attacks (higher value means better performance/robustness).

🔥 Latest News

  • Apr 28 2025: Congratulations! Our paper has been accepted for presentation at IJCAI 2025 Survey Track.
  • Mar 9 2025: 👨🏻‍💻 We are working hard to prepare the NTLBench. Thank you for your patience!
  • Feb 19 2025: ✨ We have released our survey (preprint version) on arXiv.

Survey

📌 We are actively tracking and updating the latest research about NTL.

👇 Summary of NTL methods according to Field (modal, task), Data (label space, target supervision, source data dependent), Non-Transferable Approach (feature or output space), and Robustness (whether source and target domain robustness have been evaluated). 📝 More details please refer to our paper [arXiv].

NTLBench

NTLBench is the first benchmark for non-transferable learning (NTL), which contains a standard and unified training and evaluation process. NTLBench supports 5 SOTA NTL methods, 9 datasets (more than 116 domain pairs), 5 network architectures families, and 15 post-training attacks from 3 attack settings, providing more than 40,000 experimental configurations.

Components

  • NTL Training
  • Source Domain Fine-Tuning
    • Fine-tuning Strategies (initFC-all, initFC-FC, direct-FC, direct-all)
    • TransNTL (CVPR 2024) [Paper][Code]
  • Target Domain Fine-Tuning
    • Fine-tuning Strategies (initFC-all, initFC-FC, direct-FC, direct-all)
  • Source-Free Domain Adaptation

Quickstart

1. Installation

Clone the repository and install dependencies:

git clone https://github.com/tmllab/NTLBench.git
cd NTLBench
pip install -r requirements.txt

2. Preparing Data

We currently support Digits (MNIST, USPS, SVHN, MNIST-M, SYND), RMNIST, CIFAR/STL, VisDA (VisDA-T, VisDA-V), OfficeHome, DomainNet, VLCS, PACS, and Terra Incognita.

Download datasets:

mkdir ./data/ ./saved_models/
python data_download.py --data_dir ./data/

Pre-split the dataset into training/validation/testing sets:

python data_split.py

This command will create and split datasets to the ./data_presplit folder.

✨ We will provide pre-split demo datasets in Google Drive (coming soon). You can download them and save to ./data_presplit/.

3. Training NTL

You can pre-train SL or NTL models from scratch by running:

CUDA_VISIBLE_DEVICES=0 python NTL_pretrain.py

✨ We also provide model files in Google Drive (coming soon) which were pretrained on our demo pre-split datasets. You can save them to ./saved_models/.

💡 We use wandb to organize experiments and record resutls. Config files for training NTL are stored in ./config/<domain-pair>/pretrain.yml.Important Args are illustrated as belows:

  • task_name: {SL/tNTL/tCUTI/tHNTL/tCUPI/tSOPHON} for supervised learning (SL) on the source domain, or pretraining by using different NTL methods on the source & target domains.
  • pre_split: set to True for use the pre-split data.
  • data_transform: set to ntl to follow the image transformation in NTL.
  • teacher_network:
    • VGG: vgg11, vgg13, vgg19, vgg11bn, vgg13bn, vgg19bn
    • ResNet: resnet18, resnet34, resnet50, wide_resnet50_2
    • Transformer: vit_tiny, vit_base, vit_large, vit_huge
  • teacher_pretrain: whether use the ImageNet-1K pretrained weight for initialization.
  • pretrained_teacher: always set to auto for automatic saved path and name.
  • pretrain_epochs: pretraining epoch.
  • pretrain_lr: pretrain learning rate.
  • additional parameters for HNTL: for HNTL, the above [pretrain_epochs, pretrain_lr] are invalid. We use [HNTL_Disen_epochs, HNTL_Disen_lr, ...] for training disentanglement and [HNTL_KD_epochs, HNTL_KD_lr, ...] for training KD.
  • additional parameters for tCUPI: for CUPI, to keep the same net arch as other methods, we calculate the Discrimination_loss on the Dropout output of the first Linear layer of the classifier, rather than the Bottleneck layer as original CUPI-domain implementation.

4. Attack NTL Models

4.1 Source Domain Fine-Tuning

Please run the NTL_postattack_src.py to evaluate the robustness of each NTL method. You can select the fine-tuning attack or the SOTA attack TransNTL.

CUDA_VISIBLE_DEVICES=0 python NTL_postattack_src.py

💡 Config files for Source Domain Fine-Tuning are stored in ./config/<domain-pair>/attack_src.yml. Important Args:

  • surrogate_network: please keep the same as the teacher_network. (Invalid now)
  • how_to_train_surrogate: {TransNTL/FT_Direct_ALL/FT_Direct_FC/FT_InitFC_ALL/FT_InitFC_FC}.
  • surrogate_data_percen: use which percent of training samples for attack.
  • surrogate_epochs: attack epoch.
  • surrogate_lr: attack learning rate.
4.2 Target Domain Fine-Tuning

Please run the NTL_postattack_tgt.py to evaluate the robustness of each NTL method against target domain fine-tuning. You can select the fine-tuning attack under the assumption that the attacker can access parts of labeled target domain data.

CUDA_VISIBLE_DEVICES=0 python NTL_postattack_tgt.py

💡 Config files for Target Domain Fine-Tuning are stored in ./config/<domain-pair>/attack_tgt.yml. Important Args:

  • how_to_train_surrogate: {FT_Direct_ALL/FT_Direct_FC/FT_InitFC_ALL/FT_InitFC_FC}.
4.3 Source-Free Domain Adaptation

Please also run the NTL_postattack_tgt.py to evaluate the robustness of each NTL method against SFDA methods using unlabeled target domain. You can select the fine-tuning attack under the assumption that the attacker can access parts of unlabeled target domain data.

CUDA_VISIBLE_DEVICES=0 python NTL_postattack_tgt.py

Important Args (config/*/attack_tgtun.yml or wandbsweep/attack/*tgtun*.yaml):

  • how_to_train_surrogate: {SFDA_CoWA/SFDA_DIFO/SFDA_NRC/SFDA_SHOT/SFDA_PLUE/SFDA_AdaContrast}.
  • surrogate_data_percen: use which percent of training samples for attack.
  • surrogate_epochs: attack epoch.
  • surrogate_lr: attack learning rate.

Citation

If you find this work useful in your research, please consider citing our paper:

@article{hong2025toward,
  title={Toward Robust Non-Transferable Learning: A Survey and Benchmark},
  author={Hong, Ziming and Xiang, Yongli and Liu, Tongliang},
  journal={arXiv preprint arXiv:2502.13593},
  year={2025}
}

Acknowledgement

Parts of this project were inspired by the following projects. We thank their contributors for their excellent work: