Rethinking Token Reduction for State Space Models

[Paper]

Official Implementation of EMNLP2024 Rethinking Token Reduction for State Space Models

Rethinking Token Reduction for State Space Models
Zheng Zhan*, Yushu Wu*, Zhenglun Kong*, Changdi Yang, Yifan Gong, Xuan Shen, Xue Lin, and Yanzhi Wang Northeastern University
The 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP2024)

Dependencies

# the code is tested on the environment below
pip install -r requirements.txt
pip install causal-conv1d>=1.2.0
pip install mamba-ssm==v2.0.1
pip install lm-eval==0.4.2

Evaluation

• Please refer to evaluate_mamba.sh for evaluation.
• Please refer to bench_mamba.sh for benchmarking the peak memory.
• For config related to mamba, please follow Mamba-ssm.
• For more detail, please follow Sec.5 in the paper.

Citation

If you find our paper useful or relevant to your project and research, please kindly cite our paper:

@inproceedings{zhan-etal-2024-rethinking-token,
    title = {Rethinking Token Reduction for State Space Models},
    author = {Zhan, Zheng  and Wu, Yushu  and Kong, Zhenglun  and Yang, Changdi  and Gong, Yifan  and Shen, Xuan  and Lin, Xue  and Zhao, Pu  and Wang, Yanzhi},
    editor = {Al-Onaizan, Yaser  and Bansal, Mohit  and Chen, Yun-Nung},
    booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
    month = {nov},
    year = {2024},
    address = {Miami, Florida, USA},
    publisher = {Association for Computational Linguistics},
    url = {https://aclanthology.org/2024.emnlp-main.100},
    pages = {1686--1697}
}