Hippo recall evals
May 11, 2026 · View on GitHub
Measurable recall quality. Use hippo eval <corpus.json> to run, compare
configs before/after a scoring change, or gate CI with --min-mrr.
Corpora
real-corpus.json
15 hand-written queries with expected memory IDs selected by keyword match against the current local + global store. Covers a mix of topics: project rules, dev-environment gotchas, external project references, architecture notes, and intentional near-duplicates. Some cases expect 1 memory, others expect 2-9 to exercise MMR diversity.
Regenerate after the store changes significantly:
node scripts/build-eval-corpus.mjs evals/real-corpus.json
The script looks up expected IDs by content keywords, so if the store no
longer contains a matching memory the case will be silently dropped (with a
WARNING: no matches for ... line).
Baseline results (2026-04-20, store ~1088 memories, embeddings fully populated)
Eval now routes through searchBothHybrid so global memories are in scope
(previously only local entries were searched, which capped recall on any
case with g_ expected IDs).
| Config | MRR | Recall@5 | Recall@10 | NDCG@10 |
|---|---|---|---|---|
| default (lambda=0.7, localBump=1.2) | 0.967 | 0.622 | 0.682 | 0.731 |
| --equal-sources (localBump=1.0) | 0.967 | 0.632 | 0.682 | 0.735 |
| --no-mmr | 0.967 | 0.622 | 0.682 | 0.731 |
| --mmr-lambda 0.3 | 0.967 | ~0.56 | ~0.57 | ~0.65 |
| --mmr-lambda 0.9 | 0.967 | 0.622 | 0.682 | 0.731 |
--equal-sources wins by +0.004 NDCG@10. The full gain lands on
powershell-chaining (NDCG 0.63 -> 0.69) — with the 1.2x local bump,
the one local mem_102793a1dfbb dominated the six global paraphrases.
Removing the bump lets more paraphrases into top-10.
Default stays at 1.2x for now because the general case (local has fresher
project context) still benefits, but --local-bump <f> / --equal-sources
are available as tuning knobs.
MRR = 1.0 across all configs: the first relevant hit is always ranked 1.
MMR at lambda >= 0.7 is neutral on this corpus because the cases with near-duplicates (the React trio) list all three duplicates as expected, so MMR neither helps nor hurts. MMR below 0.7 hurts because it demotes near-duplicates that are actually all relevant.
Embedding-weight sweep (0.3 to 0.9) is also neutral — the set and positional pattern of expected/non-expected in top-10 is stable regardless of the BM25/cosine blend.
Takeaway: the default config is near the corpus ceiling. To improve further we need harder cases (paraphrased queries, multi-hop links) where the right answer is not BM25-obvious. That's future work.
Methodology note
The initial version of this corpus was auto-generated by keyword match and produced Recall@10 = 0.48. After hand-pruning false-positive expected IDs (session summaries that mention the keyword in passing but aren't the canonical answer) Recall@10 jumped to 0.63. Your corpus is your eval: noisy expected lists produce noisy numbers.
Parameter sweep findings (2026-04-20)
scripts/sweep-params.mjs evals/real-corpus.json --out evals/sweep-results.md
swept 30 combinations of (MMR lambda, embedding weight, local bump) through
the keyword corpus. Full numbers in sweep-results.md; the punchline:
| MRR | R@5 | R@10 | NDCG@10 | |
|---|---|---|---|---|
| default (lambda=0.7, emb=0.6, bump=1.2) | 0.967 | 66.2% | 67.1% | 0.722 |
| best (lambda=any, emb=0.4, bump=1.2) | 1.000 | 66.2% | 67.1% | 0.745 |
| delta | +0.033 | +0.0 | +0.0 | +0.023 |
Embedding weight is the real lever. Lowering it from 0.6 to 0.4 lifts MRR to 1.0 and adds 2.3 pp to NDCG. MMR lambda had no effect on this corpus — the handcrafted cases lack near-duplicate density for MMR to de-cluster, so lambda=1.0 (relevance only) ties lambda=0.5 exactly.
We did NOT change the default because:
- This is a single corpus of 15 handcrafted, keyword-heavy queries. BM25 is well-suited to that style — lowering cosine weight makes BM25 score relatively more. Real user queries often paraphrase, where cosine wins.
- The LLM-generated corpus (
build-eval-corpus-llm.mjswith paraphrased questions) is the honest test for embedding-weight tuning. Run that before changing the default. - Overfitting to one corpus is a classic eval trap. The sweep is a tuning knob for users, not a new default.
Users who want the uplift today: hippo recall <q> --embedding-weight 0.4
or set "embeddings": {"hybridWeight": 0.4} in .hippo/config.json.
LongMemEval results (2026-04-20, full 500 questions)
Ran v0.27 (hybrid BM25+cosine + MMR + outcome boost) against the full LongMemEval 500-question benchmark for direct comparison with the documented v0.11 baseline (BM25-only).
Initial run: budget=4000 (flawed comparison)
The first run used the production default budget=4000 tokens. LongMemEval
memories average 14k chars (~3500 tokens), so the budget fit ~1 memory per
query. v0.11 used FTS5 with top_k=10 and no token budget, always
returning 10 results.
| Metric | v0.27 budget=4k | v0.11 FTS5 (top_k=10) | delta |
|---|---|---|---|
| Recall@1 | 46.6% | 50.4% | -3.8 pp |
| Recall@3 | 46.6% | 66.6% | -20.0 pp |
| Recall@5 | 46.8% | 74.0% | -27.2 pp |
| Recall@10 | 46.8% | 82.6% | -35.8 pp |
The flat R@K curve (46.6 -> 46.8) confirmed budget saturation: only ~1 result per query was returned (mean 5.7, but 72/500 queries got exactly 1).
Root cause: budget saturation, not scoring regression
The apparent 35pp regression at R@10 was caused by the benchmark runner comparing a budget-limited retrieval (v0.27) against an unlimited one (v0.11). Once budget saturation is removed, the gap shrinks to 1.6pp.
Corrected run (full 500 questions, no budget limit)
| Metric | v0.27 budget=4k | v0.27 no-budget | v0.11 FTS5 | delta (fixed) |
|---|---|---|---|---|
| Recall@1 | 46.6% | 46.6% | 50.4% | -3.8 pp |
| Recall@3 | 46.6% | 67.0% | 66.6% | +0.4 pp |
| Recall@5 | 46.8% | 73.8% | 74.0% | -0.2 pp |
| Recall@10 | 46.8% | 81.0% | 82.6% | -1.6 pp |
| answer_in_content@5 | 35.0% | 49.6% | 46.6% | +3.0 pp |
The 35pp R@10 gap was entirely budget saturation. The real scoring gap is 1.6pp at R@10. v0.27 wins on R@3 (+0.4pp) and answer_in_content@5 (+3.0pp), meaning the top-5 results more often contain the actual answer.
Per-type (full 500q):
| Type | v0.27 R@1 | v0.11 R@1 | v0.27 R@10 | v0.11 R@10 |
|---|---|---|---|---|
| single-session-assistant | 94.6% | 87.5% | 100.0% | 94.6% |
| knowledge-update | 52.6% | 70.5% | 96.2% | 93.6% |
| multi-session | 40.6% | 41.4% | 82.0% | 84.2% |
| temporal-reasoning | 42.1% | 43.6% | 78.9% | 82.7% |
| single-session-user | 34.3% | 44.3% | 71.4% | 77.1% |
| single-session-preference | 16.7% | 13.3% | 33.3% | 36.7% |
v0.27 wins on single-session-assistant (+7pp R@1, +5pp R@10) and knowledge-update R@10 (+2.6pp). v0.11 wins on single-session-user R@1 (-10pp). Overall, hybrid scoring trades some R@1 accuracy for better content relevance (answer_in_content@5 +3pp).
v1.9.0 reranker sweep (2026-05-11, full 500 questions)
F6 reranker hardening (docs/evals/2026-05-10-f6-reranker-result.md,
prereg docs/evals/2026-05-10-f6-reranker-prereg.md). Sweep ran the
v1.9.0 hybrid path against the same 500-question LongMemEval workload
with three reranker tracks plus a no-reranker baseline. This release
does not re-assert the retracted −10pp magnitude.
| Version | R@1 | R@3 | R@5 | R@10 | answer_in_content@5 | Notes |
|---|---|---|---|---|---|---|
| v1.9.0 baseline | 50.4 | 67.6 | 75.6 | 83.6 | 49.2 | no reranker; same hybrid path as v0.27 with current ingest |
| v1.9.0 features (any topK) | 51.6 | 68.2 | 75.4 | 83.4 | 48.6 | Track 1 features reranker; topK 20/50/100 byte-identical (Gate-B FAIL) |
| v1.9.0 cross_encoder | 50.4 | 67.6 | 75.6 | 83.6 | 49.2 | Track 2; identity-fallback in test env (HF blocked); not a real cross-encoder evaluation |
See docs/evals/2026-05-10-f6-reranker-result.md for the full
workload-validity verdicts, per-type breakdowns, provenance, and the
roadmap-target (R@5 ≥ 85%) framing — that target is not met on the
workload tested.
Fix: minResults option
Added minResults parameter to hybridSearch, physicsSearch, search,
and searchBothHybrid. Guarantees at least N results regardless of
token budget. Prevents budget saturation when memories are large.
# CLI usage
hippo recall "my query" --min-results 5
# Programmatic
hybridSearch(query, entries, { budget: 4000, minResults: 5 });
Production default: minResults=1 (backward compatible). Benchmark
runner default: budget=1000000, minResults=10.
Perf fixes landed while investigating
- MMR capped at top-100 (commit 014487f) -- was O(N^2) on large candidate sets. Dropped per-query time from ~50s to ~9s.
preparedCorpusoption on hybridSearch (commit c9bf1a8) -- lets batch callers skip per-query tokenization. Further drop to ~6-7s/query.
LLM-generated corpus (llm-corpus.json)
Harder-than-bootstrap cases with paraphrased queries — the right memory is not BM25-obvious, so this is where embeddings and MMR should earn their keep. Generated via Claude:
export ANTHROPIC_API_KEY=sk-...
node scripts/build-eval-corpus-llm.mjs --max 20 --out evals/llm-corpus.json
For each of --max random memories, Claude produces 3 realistic queries
that a developer might type where THAT memory is the ideal answer,
without reusing the memory's exact wording. Result: up to 3 * --max
cases with 1 expected ID each.
Rate-limited at ~1 req/s. Skip memories that fail with FAILED - <reason>.
Compare against the keyword corpus to see which effects move:
hippo eval evals/llm-corpus.json --json > llm-baseline.json
hippo eval evals/llm-corpus.json --equal-sources --compare llm-baseline.json
Model Profile Bench (null result, 2026-04-21)
Separate 20-case corpus (model-profile-bench.json) designed to test whether per-model context-injection profiles (budget + framing) could close behavioral gaps between Claude Opus 4.6 and 4.7. Tested four failure modes: invariant honor, hallucination guard, noise rejection, contradiction rejection.
Result: 0 pp gap between 4.6 and 4.7 across all four types. Both models scored 100% when hippo's default context block was injected. The planned per-model profile feature was scrapped — no measurable problem to solve.
| Model | invariant-honor | hallucination-guard | noise-rejection | contradiction-rejection |
|---|---|---|---|---|
| claude-opus-4-6 | 100% | 100% | 100%* | 100% |
| claude-opus-4-7 | 100% | 100% | 100%* | 100% |
* A single Windows shell-escaping bug in the runner (not a model failure) caused one cell to error; it was identical for both models.
Interpretation. The 4.7 "regressions" circulating on social media (long-context collapse, instruction drift, gaslighting) don't appear to affect the specific failure modes hippo guards against — at least not when 1,500 tokens of curated memory are in context. Hippo's existing context injection is sufficient; no per-model tuning needed.
Corpus + runner + judge are reusable: evals/model-profile-bench.json, scripts/run-model-profile-bench.mjs, scripts/model-profile-judge.mjs. Uses claude -p --model <id> (no API key — subscription handles cost). Full investigation: docs/plans/2026-04-21-model-aware-hippo.md and docs/plans/2026-04-21-phase-a-decision.md.
Writing new cases
A case is:
{
"id": "short-slug",
"query": "natural-language question a user might type",
"expectedIds": ["mem_abc123", "g_def456"],
"description": "optional context so a failure is self-explaining"
}
Good cases:
- Use 2-6 expected IDs for queries where MMR matters (there are near-duplicates).
- Use 1 expected ID for "is this specific memory findable?" cases.
- Phrase queries in the user's voice, not in the memory's exact wording.
Gate CI
# Fail the run if MRR regresses below 0.9
hippo eval evals/real-corpus.json --min-mrr 0.9