DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery [arxiv]

August 5, 2026 · View on GitHub

Dataset distillation aims to extract abundant information from a cumbersome original dataset into a tiny distilled dataset to reduce storage and computing consumption, while simultaneously enhancing privacy protection. Nevertheless, previous approaches typically adopt a single-stage distillation paradigm, which suffers from learning high-frequency patterns that overfit to a specific architecture, consequently suppressing the expression of semantics and leading to performance degradation across heterogeneous architectures. To address this, we propose a novel dual-stage distillation framework called DIVER, which leverages a pre-trained diffusion model to dive deeper into DIstilled data Via Expressive semantic Recovery. Semantic Inheritance filters out high-frequency "noise" and distills high-level semantic knowledge of distilled images into the latent space to retain the intrinsic semantics. Furthermore, Ssemantic Guidance enhances the preservation of original semantics by directing the sampling procedure. Ultimately, semantic fusion is designed to fuse conditional labels with inherited and guided semantics only during the concrete phase of the reverse process, compensating for the lack of category information in the original features. Extensive experiments validate the effectiveness and efficiency of our method in improving classical distillation techniques and significantly improving cross-architecture generalization, requiring processing time comparable to raw DiT on ImageNet (256x256) with only 4.02 GB of GPU memory usage.

This is an official PyTorch implementation of the paper DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery. In this work:

  • We propose a dual-stage distillation framework termed DIVER, decoupling the classic dataset distillation problem into DD and DDD.
  • We formally introduce DDD. To the best of our knowledge, this is the first work to straightforwardly distill knowledge from distilled dataset into synthetic dataset without requiring access to the original dataset.
  • We propose three recovery strategies integrated into pre-trained guided diffusion model, reviving the semantics suppressed by architecture-specific patterns in the distilled dataset without requiring additional training.
  • Extensive experimental results demonstrate that DIVER effectively and efficiently enhances cross-architecture generalization as a plugin in traditional DD, with only 2.48s per image on a single RTX-4090 GPU using 4.02GB of memory.

Synthetic Images

Usage

Requirements

torch==1.13
torchvision==0.14.0
transformers==4.27
ftfy
tensorboard
Jinja2
timm
matplotlib
diffusers==0.21.0
tqdm

conda create -n diver python=3.8
conda activate diver
pip install -r requirements.txt

How to Run

# synthesis
sh diver.sh

# evaluate
bash evaluate.sh

Bibliography

If you find this repository helpful for your project, please consider citing our work:

@article{xia2026diver,
  title={DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery},
  author={Xia, Qianxin and Shu, Zhiyong and Jiang, Wenbo and Du, Jiawei and Wang, Jielei and Lu, Guoming},
  journal={ICML},
  year={2026}
}

Reference

Our code has referred to previous work: