Pyserini: Reproducing ANCE Results

July 13, 2026 ยท View on GitHub

This guide provides instructions to reproduce the following dense retrieval work:

Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, Arnold Overwijk. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval

Note that we often observe minor differences in scores between different computing environments (e.g., Linux vs. macOS). However, the differences usually appear in the fifth digit after the decimal point, and do not appear to be a cause for concern from a reproducibility perspective. Thus, while the scoring script provides results to much higher precision, we have intentionally rounded to four digits after the decimal point.

MS MARCO Passage

ANCE retrieval with brute-force index:

python -m pyserini.search.faiss \
  --index msmarco-v1-passage.ance \
  --topics msmarco-passage-dev-subset \
  --encoder castorini/ance-msmarco-passage \
  --output runs/run.msmarco-passage.ance.tsv \
  --output-format msmarco \
  --batch-size 512 --threads 16

To evaluate:

python -m pyserini.eval.msmarco_passage_eval msmarco-passage-dev-subset \
  runs/run.msmarco-passage.ance.tsv

Results:

#####################
MRR @10: 0.3302
QueriesRanked: 6980
#####################

We can also use the official TREC evaluation tool trec_eval to compute other metrics than MRR@10. For that we first need to convert runs and qrels files to the TREC format:

python -m pyserini.eval.convert_msmarco_run_to_trec_run \
  --input runs/run.msmarco-passage.ance.tsv \
  --output runs/run.msmarco-passage.ance.trec

python -m pyserini.eval.trec_eval -c -mrecall.1000 -mmap msmarco-passage-dev-subset \
  runs/run.msmarco-passage.ance.trec

Results:

map                     all     0.3363
recall_1000             all     0.9584

MS MARCO Document

ANCE retrieval with brute-force index:

python -m pyserini.search.faiss \
  --index msmarco-v1-doc.ance-maxp \
  --topics msmarco-doc-dev \
  --encoder castorini/ance-msmarco-doc-maxp \
  --output runs/run.msmarco-doc.passage.ance-maxp.txt \
  --output-format msmarco \
  --batch-size 512 --threads 16 \
  --hits 1000 --max-passage --max-passage-hits 100

To evaluate:

python -m pyserini.eval.msmarco_doc_eval \
  --judgments msmarco-doc-dev \
  --run runs/run.msmarco-doc.passage.ance-maxp.txt

Results:

#####################
MRR @100: 0.3794
QueriesRanked: 5193
#####################

We can also use the official TREC evaluation tool trec_eval to compute other metrics than MRR@100. For that we first need to convert runs and qrels files to the TREC format:

python -m pyserini.eval.convert_msmarco_run_to_trec_run \
  --input runs/run.msmarco-doc.passage.ance-maxp.txt \
  --output runs/run.msmarco-doc.passage.ance-maxp.trec

python -m pyserini.eval.trec_eval -c -mrecall.100 -mmap msmarco-doc-dev \
  runs/run.msmarco-doc.passage.ance-maxp.trec

Results:

map                     all     0.3794
recall_100              all     0.9033

Natural Questions (NQ)

ANCE retrieval with brute-force index:

python -m pyserini.search.faiss \
  --index wikipedia-dpr-100w.ance-multi \
  --topics dpr-nq-test \
  --encoder castorini/ance-dpr-question-multi \
  --output runs/run.ance.nq-test.multi.trec \
  --batch-size 512 --threads 16

To evaluate, first convert the TREC output format to DPR's json format:

python -m pyserini.eval.convert_trec_run_to_dpr_retrieval_run \
  --topics dpr-nq-test \
  --index wikipedia-dpr \
  --input runs/run.ance.nq-test.multi.trec \
  --output runs/run.ance.nq-test.multi.json

python -m pyserini.eval.evaluate_dpr_retrieval \
  --retrieval runs/run.ance.nq-test.multi.json \
  --topk 20 100

Results:

Top20   accuracy: 0.8224
Top100  accuracy: 0.8787

Trivia QA

ANCE retrieval with brute-force index:

python -m pyserini.search.faiss \
  --index wikipedia-dpr-100w.ance-multi \
  --topics dpr-trivia-test \
  --encoder castorini/ance-dpr-question-multi \
  --output runs/run.ance.trivia-test.multi.trec \
  --batch-size 512 --threads 16

To evaluate, first convert the TREC output format to DPR's json format:

python -m pyserini.eval.convert_trec_run_to_dpr_retrieval_run \
  --topics dpr-trivia-test \
  --index wikipedia-dpr \
  --input runs/run.ance.trivia-test.multi.trec \
  --output runs/run.ance.trivia-test.multi.json

python -m pyserini.eval.evaluate_dpr_retrieval \
  --retrieval runs/run.ance.trivia-test.multi.json \
  --topk 20 100

Results:

Top20   accuracy: 0.8010
Top100  accuracy: 0.8522

Reproduction Log*