Convolutional Initialization for Data-Efficient Vision Transformers

Project Page | Paper

Jianqiao Zheng, Xueqian Li, Simon Lucey
The University of Adelaide

---

🚀 New! Explore our updated NeurIPS'25 work Structured Initialization for Vision Transformers — Released in Dec 2025

---

This is the official implementation of the paper "Convolutional Initialization for Data-Efficient Vision Transformers", including a modified version of ConvMixer and Simple ViT on CIFAR-10, CIFAR-100, SVHN and Tiny ImageNet. The code is based on vision-transformers-cifar10

Illustration of different methods to extend 1D encoding

Google Colab

If you want to try out our new initialization for ViT, check this Colab for a quick tour.

Usage

Modify convmixer.sh or vit_pex.sh first to change the data path and what experiments you want to run, and then just run

bash convmixer.sh

or

bash vit_pex.sh

Citation

@article{zheng2024convolutional,
  title={Convolutional Initialization for Data-Efficient Vision Transformers},
  author={Zheng, Jianqiao and Li, Xueqian and Lucey, Simon},
  journal={arXiv preprint arXiv:2401.12511},
  year={2024}
}