Soft Label Pruning and Quantization for Large-Scale Dataset Distillation (LPQLD)

April 18, 2026 · View on GitHub

[Paper | BibTex | Google Drive]

Official implementation of LPQLD: Soft Label Pruning and Quantization for Large-scale Dataset Distillation (IEEE TPAMI, 2026).

This work extends LPLD (NeurIPS'24) with two orthogonal improvements: Label Quantization (Q) for further storage compression, and Dynamic Knowledge Reuse (DKR) + Calibrated Student-Teacher Alignment (CA) for improved supervision diversity.

Lingao Xiao, Yang He

Abstract: Large-scale dataset distillation requires storing auxiliary soft labels that can be 30–40× (ImageNet-1K) or 200× (ImageNet-21K) larger than the condensed images, undermining the goal of dataset compression. We identify two fundamental issues: (1) insufficient image diversity and (2) insufficient supervision diversity. To address both, we propose LPQLD, which reduces soft label storage by 78× on ImageNet-1K and 500× on ImageNet-21K while improving accuracy by up to 7.2% and 2.8%, respectively. LPQLD combines Label Pruning (P) and Label Quantization (Q) for storage compression, with Dynamic Knowledge Reuse (DKR) and Calibrated Alignment (CA) for supervision diversity during training.

Installation

Download repo:

git clone https://github.com/he-y/soft-label-pruning-quantization-for-dataset-distillation LPQLD
cd LPQLD

Create environment:

conda env create -f environment.yml
conda activate lpqld

Download Datasets and Labels

Method 1: Automatic Downloading

# sh download.sh [true|false]
sh download.sh true

`true$ (\text{default}): \text{download} \text{recommended} \text{labels} \text{only} ( \times 80 \text{for} \text{ImageNet}-1\text{K}, \times 400 \text{for} \text{ImageNet}-21\text{K})
$false`: download all label variants (warning: very large, tens of GBs)

Method 2: Manual Downloading

Download labels from the tables below and place files in the following structure:

.
├── recover/
│   ├── model_with_class_bn/    ← reuse from LPLD (see below)
│   └── syn_data_LPLD/          ← reuse from LPLD (see below)
└── relabel_and_validate/
    └── syn_label_LPQLD/
        ├── FKD_cutmix_fp16_LPQLD_in1k_rn18_4k_ipc10_ratio9_topk10/
        ├── FKD_cutmix_fp16_LPQLD_in1k_rn18_4k_ipc10_ratio9_topk50/
        └── ...

Model with Class-wise BN and Distilled Images

Reuse from LPLD. Download from LPLD Google Drive: the synthesized images and class-BN models are identical.

Dataset	Model with Class-BN	Distilled Images
ImageNet-1K	50 MB	IPC10–200 (0.15–2.98 GB)
Tiny-ImageNet	81 MB	IPC50/100 (21–40 MB)
ImageNet-21K	446 MB	IPC10/20 (3–5 GB)

Previous Soft Labels vs. Ours (LPQLD)

Each cell shows storage / accuracy. Click storage size to download.

ImageNet-1K — full original labels: LPLD Google Drive

IPC	Previous (full)	LPLD ×40	LPQLD ×40	LPQLD ×80	LPQLD ×200
10	5.67 GB / 20.1%	0.14 GB / 20.2%	0.13 GB / 29.6%	0.07 GB / 27.3%	0.03 GB / 20.0%
20	11.33 GB / 33.6%	0.29 GB / 33.0%	0.25 GB / 41.2%	0.15 GB / 38.6%	0.06 GB / 30.5%
50	28.33 GB / 46.8%	0.71 GB / 46.7%	0.63 GB / 51.3%	0.37 GB / 49.6%	0.14 GB / 43.0%
100	56.66 GB / 52.8%	1.43 GB / 54.0%	1.27 GB / 56.2%	0.73 GB / 54.9%	0.29 GB / 50.1%
200	113.33 GB / 57.0%	2.85 GB / 59.6%	2.54 GB / 59.9%	1.47 GB / 58.8%	0.58 GB / 55.4%

×40 = MR-100 (top-100 logits); ×80 = MR-50 (top-50 logits); ×200 = MR-10 (top-10 logits).

ImageNet-21K — full original labels are too large to upload; compressed variants provided below.

IPC	Previous (full)	LPLD ×40	LPQLD ×40	LPQLD ×400
10	643 GB / 18.5%	16 GB / 21.3%	16 GB / 25.6%	1.6 GB / 20.9%
20	1286 GB / 20.5%	32 GB / 29.4%	32 GB / 33.8%	2.7 GB / 24.1%

×40 = label pruning only; ×400 = label pruning + quantization (MR-200).

Necessary Modification for PyTorch

Modify torch.utils.data._utils.fetch._MapDatasetFetcher to support multi-processing loading of soft label data and mix configurations:

class _MapDatasetFetcher(_BaseDatasetFetcher):
    def fetch(self, possibly_batched_index):
        if hasattr(self.dataset, "mode") and self.dataset.mode == 'fkd_load':
            if hasattr(self.dataset, "G_VBSM") and self.dataset.G_VBSM:
                pass  # G_VBSM: uses self-decoding in the training script
            elif hasattr(self.dataset, "use_batch") and self.dataset.use_batch:
                mix_index, mix_lam, mix_bbox, soft_label = self.dataset.load_batch_config_by_batch_idx(possibly_batched_index[0])
            else:
                mix_index, mix_lam, mix_bbox, soft_label = self.dataset.load_batch_config(possibly_batched_index[0])

        if self.auto_collation:
            if hasattr(self.dataset, "__getitems__") and self.dataset.__getitems__:
                data = self.dataset.__getitems__(possibly_batched_index)
            else:
                data = [self.dataset[idx] for idx in possibly_batched_index]
        else:
            data = self.dataset[possibly_batched_index]

        if hasattr(self.dataset, "mode") and self.dataset.mode == 'fkd_load':
            # NOTE: mix_index, mix_lam, mix_bbox can be None
            mix_index_cpu = mix_index.cpu() if mix_index is not None else None
            return self.collate_fn(data), mix_index_cpu, mix_lam, mix_bbox, soft_label.cpu()
        else:
            return self.collate_fn(data)

Reproduce Main Results

Step 1: Set Paths

Edit two files before running any scripts:

1. Config YAML — set the validation directory:

# relabel_and_validate/cfg/reproduce/LPQLD_in1k_[4k].yaml (and LPQLD_in21k_[2k].yaml for ImageNet-21K)
validate:
  path:
    val_dir: /path/to/imagenet/val   # ← set this

2. Reproduce script — set the distilled image directory:

# relabel_and_validate/scripts/reproduce/lpqld_in1k.sh, line 28:
train_dir="/path/to/syn_data_LPLD/LPLD_in1k_rn18_4k_ipc${ipc}"  # ← set this

# relabel_and_validate/scripts/reproduce/lpqld_in21k.sh, line 29:
train_dir="/path/to/syn_data_in21k/sre2l_in21k_rn18_2K_ipc${ipc}"  # ← set this

Step 2: Run Reproduce Scripts

cd relabel_and_validate
bash scripts/reproduce/lpqld_in1k.sh    # ImageNet-1K
bash scripts/reproduce/lpqld_in21k.sh   # ImageNet-21K

Alternatively, use downloaded labels directly — refer to README_usage.md for the three-stage pipeline.

What's New in LPQLD vs LPLD

Component	LPLD	LPQLD
Class-wise BN synthesis (§III-B)	✓	✓
Label Pruning — P (§III-C)	✓	✓
Label Quantization — Q (§III-C)	✗	✓
Dynamic Knowledge Reuse — DKR (§III-D)	✗	✓
Calibrated Student-Teacher Alignment — CA (§III-E)	✗	✓

DKR (--temp_scheduler): temperature annealing on stored pre-softmax logits extracts diverse supervisory signals across training epochs without additional storage.

CA (--temp_stu_dynamic): dynamically adjusts the student temperature as a calibrated ratio of the teacher temperature, aligning their probability distributions.

LPLD — Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?
SRe²L — Squeeze, Recover and Relabel framework
ImageNet-21K Pretraining for the Masses

Citation

@article{xiao2025lpqld,
  title   = {Soft Label Pruning and Quantization for Large-Scale Dataset Distillation},
  author  = {Lingao Xiao and Yang He},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year    = {2025}
}

@inproceedings{xiao2024lpld,
  title     = {Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?},
  author    = {Lingao Xiao and Yang He},
  booktitle = {The Thirty-eighth Annual Conference on Neural Information Processing Systems},
  year      = {2024}
}