Perseus Vault on LongMemEval (session-level retrieval, offline & judge-free)

A reproducible, fully offline measurement of how well Perseus Vault retrieves the right memory on the public LongMemEval benchmark. It reports session-level recall@k against LongMemEval's own answer_session_ids, across Perseus Vault's three search modes (fts5 keyword, dense vector, hybrid RRF). No API key, no network, no LLM. Anyone can re-run it and get the same number.

What this measures (and what it does not)

LongMemEval has two stages:

1. Retrieval — given a question and a haystack of ~48 chat sessions (~46 distractors + ~2 evidence sessions), surface the evidence. The official metric is session-level recall vs answer_session_ids. This is judge-free and deterministic. This is what this harness measures.
2. QA accuracy — feed the retrieved context to an LLM and judge the answer with another LLM. That stage needs an LLM + a judge model, so it is not offline or deterministic, and the score depends entirely on which models you pick. This harness deliberately does not produce a QA number (see "Honesty" below).

Perseus Vault's pitch is local-first, so its credibility benchmark is the half that needs no cloud: retrieval quality you can reproduce on your own machine.

Run it

# 1. Build perseus_vault (bundled embeddings are on by default)
cargo build --release

# 2. Get the real LongMemEval _s split (500 instances, ~48 sessions each, 277 MB)
curl -L https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json \
  -o longmemeval_s_cleaned.json

# 3. Run (full 500; use --max-instances N for a quick subset)
python benchmark/longmemeval/run.py --data longmemeval_s_cleaned.json

Output: a content-hashed (sha256) report.json plus a console table. The run is offline and the metrics are deterministic run-to-run across every mode: fts5 and the RRF fusion step always were (#247), and the embedding-backed dense/hybrid/auto modes are now too — the bundled ONNX backend is pinned to single-threaded, deterministic inference (#310), so the same input yields a byte-identical embedding (and therefore a byte-identical signature) on every run.

Method

• One memory per session (key = session id, body = the session's turns flattened as role: content), namespaced by question id.
• Dense vectors populated with the bundled ONNX model (all-MiniLM-L6-v2, 384-d), in-process, offline.
• Each question is queried scoped to its own haystack (via the category filter), so retrieval competes only against that instance's ~48 sessions, exactly the LongMemEval-s setting.
• Process-per-instance with a fresh DB keeps each instance's store tiny and the isolation exact.
• recall@k = the gold evidence session appears in the top k; MRR = reciprocal rank of the first gold session. Reported overall and broken down by the six LongMemEval question types.

Results

Full LongMemEval _s split: 500 questions, 23,867 sessions, offline on Windows 11 with the release binary (bundled ONNX embeddings). Fingerprinted (sha256) in report.json.

This is the default user experience after #271: a bare perseus_vault_remember then perseus_vault_recall with no manual perseus_vault_embed and no mode argument (--skip-explicit-embed --modes auto). auto exercises #271's auto-select; the run skips the explicit embed to prove #271's auto-embed-on-write is what populates the vectors.

path	recall@1	recall@3	recall@5	recall@10	MRR
keyword only (fts5)	4.2%	13.0%	23.6%	42.0%	0.126
default (auto, post-#271 + #309)	84.6%	95.2%	97.4%	99.2%	0.903
hybrid (explicit)	84.6%	95.2%	97.4%	99.2%	0.903

The headline: before #271 a bare remember+recall fell back to keyword search, which finds the right session only 4% of the time at rank 1 (LongMemEval paraphrases its questions). #271 makes auto-embed-on-write + hybrid the default, so the same bare calls now hit ~85% recall@1 / 97% recall@5 with no API key, no cloud, no LLM, and no manual step. auto == hybrid to the digit, confirming the default equals the ceiling. (Standalone dense, measured separately, is 77.0% / 93.8% — so fusing the keyword arm adds ~8 points of recall@1 over dense alone.) #309 raised the keyword arm to equal weight in the RRF fusion (it had been under-weighted at 0.5), lifting the default from 82.2% / 0.884 MRR to the numbers above. These numbers are now reproducible run-to-run across all modes (deterministic embeddings, #310).

By question type (default/auto recall@1 / recall@5):

question type	n	recall@1	recall@5
single-session-assistant	56	98.2%	98.2%
multi-session	133	90.2%	98.5%
knowledge-update	78	89.7%	98.7%
temporal-reasoning	133	83.5%	97.0%
single-session-user	70	71.4%	98.6%
single-session-preference	30	56.7%	86.7%

Equal-weight fusion (#309) improved 5 of the 6 types vs the old 0.5 weight; the small single-session-preference set (n=30) traded down (63.3→56.7 recall@1) as the net across all 500 rose. Reproduce the default experience: python benchmark/longmemeval/run.py --data longmemeval_s_cleaned.json --skip-explicit-embed --modes auto fts5 (signature 9babb85..., byte-identical run-to-run now that embeddings are deterministic, #310). Drop the flags to also measure the explicit dense/hybrid modes.

Protocol comparability and dual lanes (#1111)

comparability.py and comparability.schema.json define the provider-free perseus-vault-longmemeval-comparability/v1 manifest. A manifest binds the split/dataset commitment and type distribution, answerer/judge IDs and caps, answer/judge prompt digests, threshold semantics, retry/error policy, ingest shape, context representation, requested retrieval depth, effective delivered depth, context budgets, selection/assembly policy, evaluator denominator, exclusions/failures, and verified custody hashes.

The harness keeps two lanes separate:

• official-compatible: the official LongMemEval CoT answer prompt and per-type evaluator;
• product-optimized: a declared product answer/context/judge contract that is useful for engineering decisions but is not an official-compatible score.

compare_manifests() emits bounded field-level matches/mismatches and a like-for-like/not-like-for-like disposition. Model, prompt, judge threshold, top-k override, context, denominator, evaluator, and provenance differences are reported independently. build_dual_lane_scorecard() preserves the lanes and never emits a combined accuracy. Stale or unknown provenance, missing fields, contradictory denominators, malformed depths, and digest mismatches fail closed.

Older signed reports remain readable through read_legacy_artifact() but are explicitly legacy-readable and not relabeled or made like-for-like by inference. No provider, answerer, or judge call is made by the manifest or scorecard tests.

Reader/judge sensitivity matrix (#1138)

sensitivity_matrix.py adds a provider-free, opt-in protocol gate for separating reader, judge, prompt-lane, and retrieval-cutoff sensitivity. The strict longmemeval-sensitivity-matrix/v1 schema records, per cell:

• reader/provider identity and exact decoding parameters;
• judge/provider identity, contract, threshold, prompt digest, and whether the result is correlated with the reader;
• official-cot versus production-generic prompt lane;
• retrieval arm, requested/effective depth, context-token budget, retry policy, denominator, and artifact/configuration commitments.

The deterministic synthetic fixture can be rendered without a dataset, binary, API key, provider, answerer, judge, network call, or paid spend:

python benchmark/longmemeval/sensitivity_matrix.py \
    --fixture benchmark/longmemeval/sensitivity_matrix_fixture.json \
    --out /tmp/longmemeval_sensitivity_report.json

Every cell owns an immutable sanitized outcome set and digest. The report emits paired hash-only per-question rows, category deltas, and a sensitivity table while keeping retrieval coverage, end-to-end QA, abstention/negative behavior, judge agreement/discordance, latency, call counts, and provider usage/cost in separate fields. Missing protocol fields, incomplete cells, mismatched configuration/outcome digests, raw-payload markers, and provider activity in the provider-free mode fail closed; the writer never post-processes a baseline into a candidate cell.

A same-model answerer/judge cell is explicitly labeled correlated-same-model and cannot authorize an independent-validation claim. A later bounded paid canary is a separate operator decision: it must provide complete usage/latency/call telemetry plus a human-audit or second-judge sample before any smaller-reader or production-economics claim is eligible. This gate is off the default published claim path and does not relabel the existing official-compatible or product-optimized lanes.

Evidence sufficiency curves (#1112)

retrieval_diag.py now composes an additive sufficiency projection using longmemeval/sufficiency.py and sufficiency.schema.json. It keeps the existing single-hit/legacy coverage fields readable while adding separate curves at k=1,3,5,10,20,50 (when the retrieval depth reaches each k):

• single-hit recall / Hit@k;
• all-required-evidence coverage@k;
• latest-version coverage for rows with a declared latest valid version;
• temporal-anchor coverage for predeclared temporal rows;
• worst required rank and missing-evidence counts;
• overall, question-type, and predeclared focus-stratum curves.

The evaluator runs only after retrieval output is sealed/replayed. Gold evidence sets never enter production retrieval or context selection, and public case rows contain counts/ranks and set commitments rather than evidence IDs, prompts, or memory bodies. Missing, duplicate, unavailable, partial, truncated, and stale evidence are represented explicitly; ineligible rows are excluded from metric denominators instead of being converted to zeroes. Dataset, fixture, retrieval configuration, code, projection, and report signatures are recorded, with offline=true and zero provider/answerer/judge calls.

The existing coverage_at_k and coverage_latest_at_k fields remain for backward readability. Use --k 50 for the complete standard sufficiency ladder; smaller diagnostic depths publish only the measured prefix.

Stage 2: QA accuracy (pinned answerer + pinned judge — the vs-Zep harness, #475)

qa.py is the second stage and the head-to-head-vs-Zep harness. Per question it ingests the haystack into the real binary, hybrid-retrieves top-k sessions, answers with a pinned, named LLM (default gpt-4o-2024-08-06, temperature 0) and grades with a pinned, named judge (default gpt-4o-2024-08-06, temperature 0, strict yes/no prompt committed in qa.py). It writes a content-hashed (sha256) qa_report.json (models, split, per-category accuracy, commit, binary, verdict sha256) plus hypotheses files in LongMemEval's official format so their evaluate_qa.py can cross-check our judge. See COMPARISON.md for the comparison rules and the scoreboard vs Zep's published 63.8%.

Systems: fullcontext (all ~48 sessions), perseus_vault (top-k hybrid-retrieved sessions, the default), oracle (gold sessions only, upper bound).

Token efficiency (offline, no key needed, --dry-run). On 50 _s instances, k=5 retrieval:

system	avg sessions/q	relative context
fullcontext	48.7	8.0x (baseline)
perseus_vault (k=5)	5.0	1.0x (8.0x less)
oracle	1.0	~34x less

Perseus Vault feeds the LLM ~8x fewer tokens than dumping the whole history — and from the retrieval result above, hybrid recall@5 is 97%, so those 5 sessions contain the evidence almost every time. (Token counts use tiktoken when present, else a ~4-chars/token estimate; the ratio is tokenizer-independent. This is the honest, reproducible version of the deprecated doc's "82x fewer tokens" claim.)

Opt-in compact evidence ledger (#1108). --context-assembly evidence-ledger replaces the ranked session text with a deterministic, extractive ledger. It keeps dated source rows, turn/session provenance, user-stated evidence, and explicitly labeled assistant context; repeated normalized statements share provenance while numeric/date versions remain distinct. The default is a 12,000 estimated-token budget and the implementation rejects budgets above the 16,000-token hard cap. The default full path, none/preference guidance, and the existing ranked snippet arm are unchanged. The ledger performs no provider, judge, embedding, or summarizer call:

python benchmark/longmemeval/qa.py --data longmemeval_s_cleaned.json \
    --mock-llm --cot --context-assembly evidence-ledger --ledger-budget 12000 \
    --limit 5 --bin /path/to/perseus-vault --out ledger-smoke.json

Preregistered free gate (#1109). Before any paid counterfactual, run the provider-free paired gate over the frozen 63-case fixture. A pinned retrieval replay is preferred; without --retrieval-replay, pass --bin to reproduce retrieval locally. The report includes per-case dated evidence blocks, session inclusion, deterministic/budget/source-token checks, answer-token proxies, and causal-stratum regressions:

python benchmark/longmemeval/evidence_ledger_gate.py \
    --data /path/longmemeval_focus_canary_63.json \
    --causal-ledger /path/causal_decomposition.json \
    --focus-audit /path/focus_failure_audit.json \
    --retrieval-replay /path/focus_failure_audit.json \
    --out evidence-ledger-free-gate.json

The gate is offline by construction, requires the preregistered 18/15/6/24 strata, requires the candidate not to lose the ranked arm's gold-session inclusion, and never authorizes paid spend itself. A paid run remains a separate operator decision and must use the paired protocol from issue #1109.

Accuracy (needs a named LLM + judge; opt-in, NOT in any CI gate). To produce it:

export OPENAI_API_KEY=...        # or put the key in ~/.openai_key; OPENAI_BASE_URL optional

# Free plumbing check (no key, no network): stubbed answerer+judge, real ingest+retrieval
python benchmark/longmemeval/qa.py --mock-llm --limit 5

# Cheap paid smoke run
python benchmark/longmemeval/qa.py --limit 10

# Full 500-question head-to-head number (prints a cost estimate, requires --yes)
python benchmark/longmemeval/qa.py --yes

Defaults: answerer gpt-4o-2024-08-06, judge gpt-4o-2024-08-06, split _s, hybrid k=10 — override with --model / --judge / --split / --k / --limit (all recorded in the report). Optionally cross-check with LongMemEval's official judge by feeding the emitted hypotheses-<system>-<model>-<prompt-lane>.jsonl to their src/evaluation/evaluate_qa.py. Run every system through the same models, and report the LLM and judge by name beside the number (see COMPARISON.md).

Chain-of-thought answer prompt (#579)

LongMemEval ships two official answer prompts: a direct one (the default here) and a chain-of-thought variant (run_generation.py cot=true). The retrieval diagnostic below attributed the majority of consistent QA failures to reasoning over correctly-retrieved evidence, not retrieval — the class CoT addresses. Run the official CoT prompt with --cot:

python qa.py --data longmemeval_s_cleaned.json --cot --k 10 --yes \
    --journal cot_full.jsonl --out cot_report.json

--cot uses LongMemEval's own CoT prompt (still 100% official methodology) and raises the answer max_tokens to 1200. The answerer's complete response is passed to the official per-type judge and written unchanged to the official hypothesis artifact; it is never reduced after an Answer: marker. Both the journal _config and the report carry answer_prompt: "official-cot" (vs "plain"), hypothesis_mode: "complete-response", and the run signature includes both fields. Hypothesis filenames include the prompt lane, so plain and official-CoT outputs cannot overwrite each other. Use --only-types single-session-preference multi-session temporal-reasoning to run a weak-category slice cheaply. A single run's number is never quoted alone (see the honesty notes) — confirm with ≥2 seeds and, ideally, an independent re-grade via the authors' evaluate_qa.py on the emitted hypotheses-*.jsonl.

Retrieval coverage diagnostic (offline, judge-free — #580)

retrieval_diag.py isolates the retrieval half of QA failure with no LLM, no judge, no API cost. It replays the identical benchmark ingest + hybrid recall for every question at a deep top-K and records where each gold evidence session ranked, then reports gold-evidence coverage@k:

python retrieval_diag.py --data longmemeval_s_cleaned.json \
    --bin ../../target/release/perseus-vault --k 50 --out diag_report.json

It also works as a coverage regression guard — fail CI when recall drops:

python retrieval_diag.py --data longmemeval_s_cleaned.json \
    --bin ../../target/release/perseus-vault --min-coverage-at 20:0.95

--min-coverage-at K:FRAC exits non-zero if coverage@K falls below the floor. The report buckets misses into k-recoverable (gold ranked 11–K, a deeper k would catch them) and hard (a gold session absent from the top-K entirely — the interesting engine cases for multi-query / aggregation-aware retrieval).

The diagnostic also emits the shared provider-free replay contract beside the report:

<out stem>_replay.jsonl
<out stem>_snapshot.jsonl

Each replay envelope is perseus-vault-retrieval-replay/v1, keeps wire rank separate from final rank, preserves absent scores, and records explicit complete/partial/empty/unavailable/degraded states. The aligned snapshot is hash-only and can be passed to benchmark.package.common.replay.replay_envelope for an independent membership/order check; it contains no question text, answers, session bodies, or provider responses.

Honesty notes (read before quoting a number)

• This is a retrieval number, not end-to-end QA accuracy. Do not compare it to papers' QA-accuracy tables. Compare it only to other systems' session-level recall on LongMemEval-s.
• QA-accuracy comparisons across papers use different LLMs and judges and are not apples-to-apples. If we ever publish a QA number, it must name the exact LLM + judge and run every baseline through the identical models on the identical split.
• Perseus Vault's headline mode is hybrid (it fuses keyword + vector). Report all three modes; do not cherry-pick.
• The _s split is the retrieval-stressing one (distractors present). The oracle split contains only evidence sessions, so retrieval recall there is trivially ~1.0 and meaningless; do not benchmark retrieval on oracle.

Supersedes

This replaces the earlier benchmarks/LONG_MEM_EVAL.md, whose numbers were not reproducible (they cited a model that does not exist and mixed LLMs/judges/splits in a single comparison table). Use this harness instead.