ATM-Bench: personal memory QA, scored by the benchmark's own evaluator

August 24, 2026 · View on GitHub

The one-line summary: the official evaluator scores this run at QS 68.4264 and Recall@10 92.8924 on the full 1,013 -question split, which is first on both columns of the public board. The row is merged: pull request #2 against the leaderboard, opened 2026-08-21, was merged 2026-08-23 by the benchmark's maintainer. Two things still stop that from being a clean "state of the art" claim, and both are below: the answer model is not matched to the baselines, and the judge ran over a non-official transport.

This document is under the claim gate, so every RE-call figure in it resolves to a committed artifact and CI checks the digits against it.

There are two artifacts. The run itself is results/atm/atm_bench_full_20260821.json, which also carries the checksums of the offline package the figures come from; the answer-side decomposition in §5 is results/atm/atm_answer_diagnosis_20260822.json, regenerable at any time with benchmarks/atm_answer_diagnosis.py and zero provider calls.

Two classes of number here cannot resolve, and each says so in the source rather than passing quietly: the maintainers' own leaderboard rows in §3, which are their published figures and not a RE-call measurement, and the judge-transport comparison, whose per-question output is not retained here.

1. What ATM-Bench is

ATM-Bench (arXiv:2603.01990, leaderboard at atmbench.github.io) is a long-horizon personal referential memory QA benchmark: 1,013 questions over 11,034 personal memory items drawn from email, images and video, asked in the first person about the owner's own past ("How much did I pay in total for my accommodation for BMVC 2024?").

Two properties make it the right third opinion for a memory layer.

The corpus is a person's memory, not a document collection. Answering needs the right item out of eleven thousand near-identical receipts, itineraries and confirmations, which is the failure mode a memory layer exists to handle and which a topical IR benchmark does not test.
Half of the score never touches an LLM judge. The headline metric, QS, is three different metrics stacked by question type:

type n how it is scored
number 360 exact multiset equality against the gold value
list_recall 139 Jaccard overlap over gold evidence IDs
open_end 514 an LLM judge, officially gpt-5-mini

So 499 of 1,013 questions are scored deterministically. That half is free to re-measure and noiseless to compare, which is what made the diagnosis in §5 possible at zero API cost.

type	n	how it is scored
`number`	360	exact multiset equality against the gold value
`list_recall`	139	Jaccard overlap over gold evidence IDs
`open_end`	514	an LLM judge, officially `gpt-5-mini`

The board reports two columns: QS and Recall@10, the fraction of questions for which at least one gold evidence item is in the retrieved top 10 .

2. The result

Full split, 1,013 questions, run 2026-08-21. QS and the per-type rows are the official evaluator's atm_openai_gpt-5-mini_summary.json, unedited.

Measure	Result
Official QS	68.4264
QS, `number`	72.7778
QS, `list_recall`	59.8270
QS, `open_end`	67.7043
Recall@10	92.8924
Recall@10GT (all gold evidence in the top 10 )	86.9694
Joint@10 (QS times Recall@10)	63.5629
Questions answered	1,013 of 1,013
Blank answers	0

The evaluator's separate LLM-only summary reads 694 of 1,013 = 68.5094 . It is a diagnostic and is not substituted for the QS score, which is the board's metric.

Both retrieval figures above were recomputed for this document from retrieval.jsonl against the released ground truth, independently of the harness that produced them, and reproduce the submitted values to four decimal places.

Configuration


Embedder	`voyage:voyage-4-large`
Reranker	`voyage:rerank-2.5`
Sparse leg	lexical (Postgres FTS)
Candidate pool, then final	25 , then 10
Evidence budget	8,192 characters
Answer model	`deepseek/deepseek-v4-pro` through OpenRouter, reasoning requested `medium`
Answer accounting	1,082 provider calls, 69 truncation retries, 2,886,847 tokens
Index	isolated table and tenant; the production index was not touched

The candidate pool is 25 rather than larger because larger Voyage pools exceeded the observed rerank token-per-minute project limit, not because 25 measured best.

3. What this may and may not be compared against

The board's own rows, full split, memory and RAG systems:

type	harness	answer model	QS	Recall@10
RAG	RE-call	`DeepSeek V4 Pro`	68.4264	92.8924
Memory	Memexa (plus `Qwen3.6-27B` captions)	`DeepSeek-V4-flash`	68.04 *	79.09
Memory	Memexa	`DeepSeek-V4-flash`	65.28 *	78.93
Memory	MemPalace	`Qwen3-VL-8B-Instruct`	56.80	76.40
Memory	ScrapMem (No-Forget)	`Qwen3-VL-8B-Instruct`	52.50	70.30
RAG	ATM-RAG	`Qwen3-VL-8B-Instruct`	51.00	68.70
RAG	Self-RAG	`Qwen3-VL-8B-Instruct`	50.30	68.70
Memory	Mem0	`Qwen3-VL-8B-Instruct`	43.50	61.90

* the board marks Memexa's QS as measured with a DeepSeek-V4-flash judge rather than gpt-5-mini, so it is shown for reference and is not directly comparable. Its Recall is.

⛔ Three boundaries, and none of them is a footnote.

The answer model is not matched. Most baselines answer with Qwen3-VL-8B-Instruct; this run answers with DeepSeek V4 Pro. So the QS column compares systems as configured, not retrieval quality holding the reader fixed, and a share of the QS lead is bought by a stronger answerer. The board's own Oracle rows price that share: handed the gold evidence with no retrieval at all, Qwen3-VL-8B-Instruct scores 78.19 and the strongest listed answerer scores 86.00 . A controlled claim needs this run repeated on the baselines' answerer, and that has not been done.
Recall@10 is the clean column. It depends on neither the judge nor the answer model, so 92.8924 against 79.09 is like-for-like in a way the QS comparison is not. It is also the result this project would defend first, because retrieval is what RE-call is.
The judge ran over a non-official transport. The official evaluator, its prompt and the gpt-5-mini judge identity were all kept; only the HTTP endpoint was OpenRouter rather than OpenAI directly. 🔁 Ruled on 2026-08-23: the maintainers accepted it, and the merged row carries a source comment recording the disclosure. This paragraph stays because the ruling is about admissibility, not about fidelity, and the measurement below is what bounds the latter. A same-prompt comparison of the two routes on 60 already-judged open_end questions agreed on 56 of 59 verdicts and ran about 1.7 points lower on that type, which is roughly 0.86 QS at its population share. That spread cannot be separated from the judge's own run-to-run variance, so treat it as an upper bound on route fidelity rather than a correction.

The submission is merged, not merely open: pull request #2 against the leaderboard, opened 2026-08-21, was merged 2026-08-23 by JingbiaoMei, one of the benchmark's maintainers. "First on the board" is now a placement rather than only a claim about arithmetic, still subject to the three comparability limits above.

Two smaller scope limits: the 31 -question ATM-Bench-Hard split was not run, and index_time is not reported because the run reused an existing index rather than building one.

4. Retrieval is not the bottleneck, and the gap says so

type	Recall@10	Recall@10GT	QS	gap
`number`	93.0556	86.6667	72.7778	20.2778
`list_recall`	89.9281	83.4532	59.8270	30.1011
`open_end`	93.5798	88.1323	67.7043	25.8755
overall	92.8924	86.9694	68.4264	24.4660

Retrieval finds the evidence for roughly nine questions in ten and the score lands at roughly seven in ten. Whatever is left is answer selection, synthesis and formatting, not search. list_recall is the weakest arm by a wide margin despite retrieval on it being only three points below the others.

5. Where the remaining loss actually is

Measured 2026-08-22 by replaying the archived run package against the official scorer, zero API calls. Reproducible with benchmarks/atm_answer_diagnosis.py, which reproduces QS 0.6843 and every per-type accuracy exactly and refuses to write its artifact if it does not, so the apparatus is verified before anything is concluded from it. The abstention and tokenisation rules are imported from the evaluator's own normalizer rather than reimplemented.

22.53 QS points are lost on questions where the complete gold evidence was already in the top 10 . Split by type: number 5.92 , list_recall 3.58 , open_end 13.03 .

The largest single recoverable mechanism is over-abstention:

The model declines to answer 156 of 1,013 questions (15.4 %). Only 17 of those are correct, and every one of the 23 gold-abstention questions is open_end.
The other 139 wrong refusals are 13.72 QS of dead loss, and 92 of them had the complete gold evidence retrieved.
Because no gold abstention exists outside open_end, a rescue restricted to number and list_recall (53 and 31 refusals, each scoring exactly 0.0000 ) cannot lose points.

⛔ The calibrated retrieval score does not separate a correct refusal from a wrong one. P(correct-abstention top-1 score < wrong-abstention top-1 score) = 0.493 , where a signal carrying no information sits at one half. The medians are 0.5192 against 0.5229 . So "gate abstention on the trust layer" is dead as stated: a threshold on that signal is a rigged coin, not a judgement. A genuine sufficiency classifier is required, or nothing.

The second mechanism is selection among retrieved items: 117 questions, 11.31 QS, were answered wrongly with at least 80 % of the gold answer's content tokens present in the evidence the model actually received. A representative case named "Cambridge, U.S." and was answered with a Cambridge UK address while both were on screen.

And there is a floor. On 18 questions, 1.78 QS, the gold item was retrieved but its text description does not contain the answer, which is the modality ceiling of answering from generated descriptions rather than from the image. That part is not winnable by any prompt or selector.

🔁 Corrected 2026-08-22: that floor was published here as twenty questions and 1.97 QS. The throwaway script it came from tested (coverage or 0) < 0.5, so two questions whose gold answer has no content tokens, and whose coverage is therefore unmeasurable, became a coverage of zero and were counted as "the answer was not on screen". token_coverage now returns None rather than 0.0 for that case and the filter honours it. Recorded rather than silently fixed because the direction is the informative part: treating a missing input as a clean zero inflated a ceiling, which is the one direction that makes a problem look less winnable than it is.

Five answer-side arms have already been measured on a 300 -question subset and four had negative point estimates: answer-format contracts, per-item qualifier marking, a disposition sentence, and evidence budget reallocation. The best arm was the official prompt with the original greedy packer, and nothing beat it with a confidence interval excluding zero. Do not repeat that family without a new mechanism. The pre-registrations and their appended results are on claude/atm-answer-selection-public.

6. Reproduction, and what is left of the gap in it

The offline run package (manifest.json, answers.jsonl, retrieval.jsonl and the four official evaluator outputs) is archived outside this tree, and its SHA-256 checksums, the dataset hashes and the evaluator file hash are recorded inside results/atm/atm_bench_full_20260821.json. The evaluator is ATM-Bench's own memqa/utils/evaluator/evaluate_qa.py, run with --metrics atm.

The harness is now in this repository, byte for byte

Landed 2026-08-23. Both files are copied from run commit 6c0ec26b without a character changed, and their hashes are recorded in results/atm/atm_harness_20260823.json:

File	What it does	`SHA-256` prefix
`benchmarks/atm_full_run.py`	retrieve, rerank, answer, checkpoint; produced `answers.jsonl` and `retrieval.jsonl`	`d15cc745790a9d23`
`benchmarks/atm_bench.py`	the retrieval-only driver, and the corpus builder the runner imports	`88cde31a0e50256c`

They are frozen rather than maintained, and the freeze is mechanical rather than a note: the pin in tests/test_atm_runner_published.py fails on a single changed byte, and both files are exempt from ruff and mypy because a style fix is a falsification of this pointer and not an improvement. The eight lint findings and six type findings they carry are listed, with their reasons, beside the exemptions in pyproject.toml.

One library change was required, and it says something about the run

The frozen runner records diagnostics.max_dense_score for every question. That field was added on the run's own private branch and had never existed on master, so publishing the runner alone would have published a program that raises AttributeError on its first retrieval record. It is ported back in the same change, populated at both of the retriever's construction sites, and pinned by a test that a None default cannot satisfy.

The general point is worth more than the field: freezing a file does not freeze the library it calls, and the reproduction pointer is only as good as the binding between them.

What still stands between this and a byte-exact re-execution

The library moved. Between the run commit and this publication, recall/ changed by 49 files, 14,462 insertions and 344 deletions. So a re-execution today runs the same driver over a different recall, which reproduces the method and is not expected to reproduce the numbers to the last decimal.
The answer model is a moving alias. deepseek/deepseek-v4-pro on 2026-08-21 is not addressable by name today, and the same limit applies to the judge.
The dataset is an input, not a file here. ATM-Bench's own corpus and question files stay outside this repository, as every benchmark's do; their SHA-256 sums are in the run artifact.

What has closed is the part that was ours to close: the code is public, it is provably the code that ran, and the claim can be checked rather than taken on trust.

python -m pytest tests/test_atm_runner_published.py -q

7. Prior context in this repository

Document	What it adds
MTRAG_BENCHMARK.md	The other external benchmark, where RE-call does not top the board, and the abstention result that does hold up
EVIDENCE.md	The one-line version of this result beside every other claim and its limit
../results/ARTIFACTS.md	The artifact map, including this run's checksums