Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning

Prerequisites:

pip install -r requirements.txt

Sparse MeZO Training

# for llama-7B training with Sparse MeZO
bash run.sh

Noted that for vanilla MeZO training, we can directly change ratio to 1.0 in run.sh.

Acknowledgements

Our code is based on the official repo of MeZO.

Ciatation

@inproceedings{
liu2025sparse,
title={Sparse Me{ZO}: Less Parameters for Better Performance in Zeroth-Order {LLM} Fine-Tuning},
author={Yong Liu and Zirui Zhu and Chaoyu Gong and Minhao Cheng and Cho-Jui Hsieh and Yang You},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=Tjw0ACu3NL}
}