LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking

April 22, 2024 · View on GitHub

:grapes: [Read our arXiv Paper]  

:bulb: Notice

  • Applying LORS to AdaMixer. The code is implemented as a python class called FactorizedAdaptiveMixing, located in the file AdaMixer_with_LORS/mmdet/models/roi_heads/bbox_heads/factorized_adaptive_mixing_operator.py.
  • Applying LORS to Transformer(Deit-Tiny). The code consists of python class MultiheadAttention_LORS and FFN_LORS, implemented in the file DeiT-Tiny_with_LORS/mmcls/models/backbones/lors_vit.py.

:fire: News

  • 2024.04.22 Release training and inference codes.
  • 2024.04.05 LORS is selected as a highlight poster.
  • 2024.02.27 LORS is accepted by CVPR 2024.