LiT5 (List-in-T5) Reranking

August 9, 2024 · View on GitHub

RankLLM

We have integrated LiT5 into RankLLM, which is actively maintained and includes additional improvements. We highly recommend using RankLLM.

📟 Instructions

We provide the scripts and data necessary to reproduce reranking results for LiT5-Distill and LiT5-Score on DL19 and DL20 for BM25 and SPLADE++ ED first-stage retrieval. Note you may need to change the batchsize depending on your VRAM. We have observed that results may change slightly when the batchsize is changed. This is a known issue when running inference in bfloat16. Additionally, you may need to remove the --bfloat16 option from the scripts if your GPU does not support it.

Note, the v2 LiT5-Distill models support reranking up to 100 passages at once.

Models

The following is a table of our models hosted on HuggingFace:

Model NameHugging Face Identifier/Link
LiT5-Distill-basecastorini/LiT5-Distill-base
LiT5-Distill-largecastorini/LiT5-Distill-large
LiT5-Distill-xlcastorini/LiT5-Distill-xl
LiT5-Distill-base-v2castorini/LiT5-Distill-base-v2
LiT5-Distill-large-v2castorini/LiT5-Distill-large-v2
LiT5-Distill-xl-v2castorini/LiT5-Distill-xl-v2
LiT5-Score-basecastorini/LiT5-Score-base
LiT5-Score-largecastorini/LiT5-Score-large
LiT5-Score-xlcastorini/LiT5-Score-xl

Expected Results

This table shows the expected results for reranking with BM25 first-stage retrieval

DL19

Model NamenDCG@10
LiT5-Distill-base71.7
LiT5-Distill-large72.7
LiT5-Distill-xl72.3
LiT5-Distill-base-v271.7
LiT5-Distill-large-v273.3
LiT5-Distill-xl-v273.0
LiT5-Score-base68.9
LiT5-Score-large72.0
LiT5-Score-xl70.0

DL20

Model NamenDCG@10
LiT5-Distill-base68.0
LiT5-Distill-large70.0
LiT5-Distill-xl71.8
LiT5-Distill-base-v266.7
LiT5-Distill-large-v269.8
LiT5-Distill-xl-v273.7
LiT5-Score-base66.2
LiT5-Score-large67.8
LiT5-Score-xl65.7

This table shows the expected results for reranking with SPLADE++ ED first-stage retrieval

DL19

Model NamenDCG@10
LiT5-Distill-base74.6
LiT5-Distill-large76.8
LiT5-Distill-xl76.8
LiT5-Distill-base-v278.3
LiT5-Distill-large-v280.0
LiT5-Distill-xl-v278.5
LiT5-Score-base68.4
LiT5-Score-large68.7
LiT5-Score-xl69.0

DL20

Model NamenDCG@10
LiT5-Distill-base74.1
LiT5-Distill-large76.5
LiT5-Distill-xl76.7
LiT5-Distill-base-v275.1
LiT5-Distill-large-v276.6
LiT5-Distill-xl-v280.4
LiT5-Score-base68.5
LiT5-Score-large73.1
LiT5-Score-xl71.0

✨ References

If you use LiT5, please cite the following paper: [2312.16098] Scaling Down, LiTting Up: Efficient Zero-Shot Listwise Reranking with Seq2seq Encoder-Decoder Models

@ARTICLE{tamber2023scaling,
  title   = {Scaling Down, LiTting Up: Efficient Zero-Shot Listwise Reranking with Seq2seq Encoder-Decoder Models},
  author  = {Manveer Singh Tamber and Ronak Pradeep and Jimmy Lin},
  year    = {2023},
  journal = {arXiv preprint arXiv: 2312.16098}
}

🙏 Acknowledgments

This repository borrows code from the original FiD repository, the atlas repository, and the RankLLM repository!