eval/

August 17, 2026 · View on GitHub

The gate for Documentation/research_roadmap.md. Nothing in Phase 1 or Phase 2 lands without a measured delta against these numbers. The numbers themselves live in BASELINE.md.

Everything here is read-only with respect to the product: no file under rag_system/, backend/ or src/ is modified or imported-and-monkeypatched. The harness calls the shipped pipeline objects directly.

Layout

Path	What it is
`corpora/atlas7_service_manual.pdf`	2-page planted-fact PDF (fictional espresso machine). 21 facts.
`corpora/northwind_leave_policy.pdf`	3-page synthetic HR handbook, generated by `make_hr_handbook.py`. 24 facts.
`corpora/*.facts.json`	Sidecars: `{id, topic, expected, summary}` per planted fact. `expected` is the verbatim answer-bearing substring.
`corpora/repo_docs.facts.json`	31 prose anchors into `Documentation/*.md` — the real, heterogeneous, third corpus. Referenced in place, never copied.
`corpora/acquisition/*.pdf`	10 interlinked synthetic M&A documents (2 pages each) — the cross-reference corpus, added for roadmap Phase 4. Reused verbatim from PromtEngineer/agentic-file-search `data/test_acquisition/`.
`corpora/acquisition.facts.json`	100 planted facts plus a `cross_references` block: the 54 `Document:` / `Exhibit` / `Schedule` pointers between those PDFs, `to: null` for the 2 deliberately dangling ones.
`corpora/rfc/*.txt`	23 interlinked IETF RFCs (QUIC / HTTP-3 family), byte-for-byte from rfc-editor.org — the only corpus whose naming and referencing conventions this project did not author. `corpora/rfc/download.py` re-fetches them and checks the link graph; `corpora/rfc/MANIFEST.md` is the rationale. Sidecar: `corpora/rfc/rfc.facts.json` (26 facts).
`corpora/verify_facts.py`	Gate 1: asserts every `expected` string exists in its source document, and every cross-reference cue in its `from` document. Recurses into subdirectories, so the `rfc` sidecar is covered too.
`corpora/make_hr_handbook.py`	Regenerates the HR PDF and re-checks its sidecar.
`build_goldset.py`	Reverse-generates one query per dimension tuple with `qwen3.5:4b`. One-shot. Covers the three Phase 0 corpora only — `acquisition.jsonl` and `rfc.jsonl` are hand-authored.
`finalize_goldset.py`	Applies the recorded human verification pass; writes the committed gold set.
`verify_crossref_goldset.py`	Row-level gate for `goldset/acquisition.jsonl`: answer in its named document, no verbatim leak into the query, each answer string unique to one document, `requires_crossref` / `multi_document` consistent with `anchor_doc`, and `anchor_doc` itself a corpus document.
`verify_rfc_goldset.py`	The same row-level gate ported to `goldset/rfc.jsonl` (normalisation handles the RFCs' 72-column hard wrapping).
`goldset/<corpus>.jsonl`	The gold set of record. 120 rows: 24 each for `atlas7`, `hr`, `docs`, `acquisition` and `rfc`. Plus `multiturn.jsonl` (12 hand-authored multi-turn rows) — no runner is wired for it yet.
`goldset/_generated/*.raw.jsonl`	Raw model output, kept for audit.
`run_eval.py`	Retrieval metrics: recall@5/10/20 first stage, nDCG@10 before and after rerank, the `*_final` family over the list the answer stage actually sees (post-rerank, post-crossref-hop), the `requires_crossref` slices, per-query latency, and the `final == first_stage` invariant check whenever nothing is allowed to reorder or append.
`judge.py`	Binary groundedness judge + its own validation harness.
`judge_validation.jsonl`	20 hand-built cases, 10 grounded / 10 subtly ungrounded.
`judge_hard_cases.jsonl`	18 real system answers, hand-adjudicated — the judge screen that decides judge swaps.
`validate_judge_hard.py`	Scores a candidate judge against `judge_hard_cases.jsonl`; majority vote over an odd number of runs, parse errors reported as errors rather than coerced to votes.
`smoke_e2e.py`	Starts both services on temp storage, indexes the Atlas-7 PDF over HTTP, asserts answers/citations/persistence, tears down.
`.eval_indexes/`	Cached LanceDB indexes, keyed by embedder. Git-ignored, safe to delete.
`results/`	Run outputs (JSON + log). Git-ignored.

Running it

All commands are from the repo root, with the project venv.

# gate 1 — planted facts really are in the source documents
.venv/bin/python eval/corpora/verify_facts.py

# gate 2 + retrieval metrics on the shipped defaults (reranking ON with
# threshold selection since arm G — that is what ships, so that is what a bare
# run measures)
.venv/bin/python eval/run_eval.py --corpus all \
  --json-out eval/results/shipped_defaults.json

# first-stage-only control arm
.venv/bin/python eval/run_eval.py --corpus all --no-rerank \
  --json-out eval/results/shipped_defaults_norerank.json

# with a different reranker, for the reranker A/B — naming one swaps the model
.venv/bin/python eval/run_eval.py --corpus all \
  --reranker Qwen/Qwen3-Reranker-4B \
  --json-out eval/results/rerank_ab.json

# judge validation (TPR / TNR / confusion matrix)
.venv/bin/python eval/judge.py --validate

# end-to-end, starts :8000 and :8001 as children on a temp DB, then kills them
.venv/bin/python eval/smoke_e2e.py

run_eval.py flags: --embedder, --reranker, --corpus {atlas7,hr,docs,mixed,acq,acq+docs,rfc,all}, --k, --chunk-size, --no-rerank, --retry {profile,on,off}, --crossref-hop {profile,on,off}, --overview-prefilter {profile,off,boost,restrict}, --overviews {off,on}, --decompose, --aggregate {max,mean}, --coverage-only, --force-reindex, --json-out, --verbose.

acq and acq+docs are the Phase 4 corpora (see below); rfc is the real-document shakedown corpus (decisions/rfc-shakedown-2026-08-13.md). mixed deliberately does not include any of them: it is the corpus every Phase 0/1/2 number is quoted against and it has to keep meaning the same thing.

Since roadmap Phase 2 the harness drives RetrievalPipeline.retrieve_candidates() rather than calling MultiVectorRetriever.retrieve() itself, so first stage, reranking and the evidence-sufficiency retry are the shipped code path. Two consequences worth knowing:

--retry defaults to profile, i.e. on, matching what ships. It is the one stage in the list below that the harness does not disable, because it is conditional: it only fires on queries whose first pass found weak evidence. Use --retry off for the control arm of an A/B. When it fires it makes one enrichment-model call, so a bare run is no longer fully deterministic — the firing set is deterministic, the rewrites are not.
first_stage_ms now covers first stage + rerank + any retry on that query, and rerank_ms is reported as 0 on this path. Per-query timings are no longer split by stage.

--decompose runs QueryDecomposer once per gold row (cached under eval/.eval_indexes/_subqueries/, keyed by corpus + decomposer model + prompt version, so the on and off arms differ only in whether the sub-queries are used) and hands them to the rerank stage. The first stage always uses the full original query — that is the whole of roadmap item 2.2. With the rerank stage off there is no consumer for sub-queries, so --decompose is skipped entirely — literally a no-op, not even the LLM calls.

--embedder defaults to whatever EMBEDDING_MODEL resolves to in rag_system/main.py, i.e. the shipped default microsoft/harrier-oss-v1-0.6b unless the env var overrides it. Each embedder gets its own index directory (eval/.eval_indexes/<slug>/) and the embedder is part of the cache fingerprint, so two embedders can neither share nor inherit an index.

The rerank stage follows the shipped profile, which has it on since arm G (2026-08-14): Qwen/Qwen3-Reranker-4B with min_score: 0.5 / min_keep: 3 / top_k: 10 threshold selection (DECISIONS.md records the earlier off-by-default call it supersedes). When the stage is on the harness keeps that selection block and overrides only the model name, so the final metrics describe the list the answer stage actually sees — not the reorder-without-selection stack a bare block would measure. --reranker <model> swaps the model; --no-rerank forces the stage off.

Regenerating the gold set (only when the corpora or the dimension table change):

.venv/bin/python eval/build_goldset.py --corpus all   # LLM writes the questions
.venv/bin/python eval/finalize_goldset.py             # applies the recorded verification pass

How the gold set is built

Reverse-generated the structured way, not "give me some questions":

Dimension tuples are hand-authored in build_goldset.py: (anchor fact(s)) × (question type: factoid / procedural / comparative / negative) × (difficulty: easy / hard). 24 tuples per corpus.
- negative means a question about a restriction, exclusion, limit or invalidating condition that the document does state — not a question the document cannot answer. Every gold row is answerable.
- easy reuses the document's vocabulary; hard paraphrases away from it, so the lexical leg cannot carry the query.
qwen3.5:4b phrases the question for each tuple, through the repo's own OllamaClient. The label — the answer-bearing substring — is fixed by the tuple, never by the model.
Every pair is verified by hand (see finalize_goldset.py, which records the verdict and the reason for each row, and goldset/*.jsonl, which carries them per row under verification).
Gate 1 (verify_facts.py): the expected string is in the source document.
Gate 2 (run_eval.py, run automatically before scoring): the expected string survives conversion + chunking into at least one indexed chunk. Rows that fail are printed and land in coverage_failures in the results JSON — they are never silently dropped.

Gold relevance is answer-bearing text, not chunk ids, so the set survives re-chunking, a different chunk size, and an embedder swap. That is the whole reason it can gate Phase 1's A/B tests.

The `acquisition` gold set is different

goldset/acquisition.jsonl is hand-authored, not model-generated, because its whole purpose is a property no dimension tuple can express: the answer is in a different document from the one the question points at. Eight of its 24 rows adapt questions from the source repo's TEST_QUESTIONS.md; the rest are new. Its rows carry two extra fields on top of the shared schema:

Field	Meaning
`dimensions.requires_crossref`	bool. `true` when the query's premise names or paraphrases document A while at least one `expected` string lives in document B, reachable from A only through an explicit reference. This slice is roadmap item 4.2's metric.
`expected_sources`, `anchor_doc`, `multi_document`	The document holding each `expected` string, the document the query's premise points at (`null` when the query names none), and whether the row's answer spans more than one document. `requires_crossref` is exactly `anchor_doc is not None and any(source != anchor_doc)`.

Every row was verified mechanically by verify_crossref_goldset.py, and the counts are recorded in BASELINE.md § Phase 4 baseline: the expected string is in its named document, the query does not contain it verbatim, and the string occurs in exactly one of the ten documents — without that last check "the answer is in another document" would not be a claim you could measure. Re-run it any time:

.venv/bin/python eval/verify_crossref_goldset.py

by_dimension therefore gains requires_crossref=true / =false buckets, and summary.<corpus>.crossref / .crossref_control carry the same slice per corpus (pooling across corpora would double-count rows that appear in both acq and acq+docs). Rows without the key — every row outside acquisition — are skipped, so no pre-existing slice moves.

Metric definitions

recall@k — over the first-stage (pre-rerank) ranking. match: "any" rows hit when one of the top-k chunks contains an expected string; match: "all" rows (the comparatives) hit only when the top-k union covers every expected string. Mean over queries.
nDCG@10 — binary per-chunk relevance (chunk contains any expected string), DCG over the top 10 divided by the IDCG of the same candidate set sorted ideally. A query whose candidates contain nothing relevant scores 0, so a first-stage miss is never hidden by the ranking metric.
Consequence worth knowing: a match: "all" query that retrieved only one of its two anchors scores recall = 0 but can still score nDCG@10 = 1.0 — ranking was perfect, coverage was not. Read the two columns together.
Latency is wall-clock per query, split into first stage and rerank, on the machine in BASELINE.md. It is not a benchmark of anything but this laptop.

What the harness turns off, and why

run_eval.py starts from PIPELINE_CONFIGS["default"] and disables:

Stage	Why
Contextual enrichment	One LLM round-trip per chunk. Nondeterministic, and it changes the indexed text, which would make the substring labels ambiguous.
Document overviews	One LLM round-trip per document; only feeds triage, which the harness does not exercise.
Late chunking	Doubles the vectors and merges sibling text into hits, which would smear the substring labels across chunk boundaries.
Context expansion	Same smearing problem; and when the reranker runs it is a no-op anyway (see `retrieval_pipeline.md`).
Query decomposition, verification, synthesis	Downstream of the two metrics being measured, and each adds LLM calls. Decomposition can be switched back on for the item-2.2 A/B with `--decompose`.

The evidence-sufficiency retry is deliberately not in that list — see the --retry note above.

chunk_size is 512, matching what the HTTP path sends (api_server.py), not the 1500-token CLI default.

Caveats that the numbers do not show on their own

atlas7, hr and acq in isolation are saturated. They are 1, 2 and 13 chunks, so k=20 sweeps the entire corpus and recall@k is 1.0 by construction. Their isolated rows are a smoke check on the plumbing, not a retrieval measurement. mixed is the row to track for Phase 0–2, acq+docs for Phase 4.
acq+docs is a weak stress test, honestly labelled. Its 360 distractor chunks are localGPT documentation — topically disjoint from an M&A deal room, so they compete far less than same-domain distractors would. Read the crossref slice with the caveats in BASELINE.md § The crossref slice is not weak.
The docs corpus is live repo content. Editing Documentation/*.md changes the corpus, invalidates the cached index, and moves the baseline. improvement_plan.md and research_roadmap.md are excluded for exactly this reason (DOCS_EXCLUDE in run_eval.py) — they are the files this harness's own bookkeeping edits.
Anchors are reused across dimensions. The two PDFs have ~20 facts each and carry 24 queries each, so a few facts back two differently-typed questions.
The judge is an LLM. Its TPR/TNR are measured on 20 hand-built cases, which is a small sample; treat the interval, not the point estimate, as the truth.

Benchmarking roadmap Phase 4

Phase 4 (Documentation/research_roadmap.md § Ideas adopted from agentic-file-search) is implemented behind profile flags that ship OFF — retrieval.crossref_hop, retrieval.overview_prefilter and retrieval.document_escalation are all enabled: False in the default profile. The harness drives them with --crossref-hop, --overview-prefilter and --overviews; this section is the protocol the A/Bs follow, written before the code so the comparison cannot be retro-fitted to a result. The pre-implementation numbers are recorded in BASELINE.md § Phase 4 baseline (pre-implementation); every "off" arm below should reproduce them.

Ground rules, the same three as every other gate in this harness:

Only acq and acq+docs move. If a Phase 4 flag changes mixed, that is a regression to explain, not a result to report. Re-run --corpus mixed on the same tree for both arms — the docs corpus is live repo content, so comparing against a number measured on a different tree proves nothing.
One flag at a time, both arms in the same session, same embedder, same k, same chunk_size. --force-reindex whenever the flag touches index content or chunk metadata (4.2 and 4.3 do).
Report the requires_crossref=true slice next to its control, never alone: the whole claim of item 4.2 is a gap between the two, and both move when the retriever changes.

4.2 — `retrieval.crossref_hop`

The headline Phase 4 A/B, and the one the acquisition corpus exists for.

The switch is --crossref-hop {profile,on,off}: apply_phase4_settings() writes retrieval.crossref_hop.enabled into the config exactly the way apply_retry_setting() writes the retry block — in run_eval.py, not as an env var, and not by editing the shipped profile. Then:

# off (must reproduce BASELINE.md § Phase 4)
.venv/bin/python eval/run_eval.py --corpus acq+docs --crossref-hop off --force-reindex \
  --json-out eval/results/p4_crossref_off.json
# on
.venv/bin/python eval/run_eval.py --corpus acq+docs --crossref-hop on --force-reindex \
  --json-out eval/results/p4_crossref_on.json

Read, in this order: summary["acq+docs"]["crossref"]["ndcg@10_first_stage"] against ["crossref_control"] (0.748 vs 0.796 today), then the crossref recall vector, then mixed for collateral damage. Because the hop adds candidates, also check that candidates per query has not silently grown past --k — a recall win bought by retrieving more chunks is not a ranking win. The k = 3 column in the baseline table is the sharpest available comparison: at k = 3 the crossref slice is at 1.000 recall today, so a hop that pays for itself has to show up as nDCG, not recall.

4.3 — `retrieval.overview_prefilter`

The only Phase 4 item that cannot reuse the cached indexes: the harness disables document overviews unless --overviews on is given, and the prefilter needs them. Both arms must therefore be run with overviews on, which costs one LLM call per document at index time and makes the index build nondeterministic. --overviews on builds into a separate _ov index directory and changes the fingerprint, so it cannot clobber the tracked indexes.

# both arms need overviews ON; --overview-prefilter picks the arm
.venv/bin/python eval/run_eval.py --corpus acq+docs --overviews on --force-reindex \
  --overview-prefilter off --json-out eval/results/p4_overview_off.json
.venv/bin/python eval/run_eval.py --corpus acq+docs --overviews on --force-reindex \
  --overview-prefilter boost --json-out eval/results/p4_overview_on.json

Do not compare an overview-prefilter arm against the numbers in BASELINE.md: those indexes have no overviews in them. The control arm must be re-measured. acq+docs is the corpus that can show anything here — acq alone is 10 documents and 13 chunks, so "restrict to the top documents" has almost nothing to restrict.

4.1 — `retrieval.document_escalation`

Not a first-stage retrieval metric. Escalation reassembles a document for synthesis after the 2.1 retry still lands weak, so run_eval.py cannot see it: the harness stops at the ranked chunk list and never synthesises. Two things it can contribute, and they are worth logging:

The trigger set. The retry already reports per query (summary.retry_fired / retry_fire_rate / retry_kept; today 1/24 on acq, 8/48 on acq+docs, and 32/48 at --k 5). Whatever condition 4.1 escalates on is a subset of that, so --retry on vs --retry off bounds how often escalation can fire before anyone writes it.
Everything else belongs in smoke_e2e.py and judge.py — answer contains the fact, citation names the right document, groundedness verdict — driven over HTTP against a session indexed on eval/corpora/acquisition/. That is the arm that measures 4.1, and it is answer quality, not recall.

4.4 — filter DSL (`filters` on /chat and /chat/stream)

Measurable here only once run_eval.py can pass a filters argument through retrieve_candidates(). The natural A/B on this corpus is a per-row filters field in the gold set (e.g. document = "07_nda.pdf") on the rows whose answer document is unambiguous, then:

.venv/bin/python eval/run_eval.py --corpus acq+docs \
  --json-out eval/results/p4_filters_off.json
.venv/bin/python eval/run_eval.py --corpus acq+docs --filters \
  --json-out eval/results/p4_filters_on.json

The number that matters is nDCG@10 on the control slice, not the crossref slice: a document filter derived from the query's anchor document is exactly the wrong thing to apply to a crossref row, and a filter A/B that improves the control while wrecking requires_crossref=true is the expected — and reportable — outcome.

4.5 — per-query token/cost tracking

Nothing to A/B: it adds fields to the SSE complete event, it does not change retrieval. Assert the fields exist and are non-zero in smoke_e2e.py; do not give it a row in a retrieval table.

4.6 — `ask <folder>` ephemeral mode

Also not a run_eval.py job — it is a CLI entry point that builds a throwaway index. The check it needs is an equivalence one: python -m rag_system.main ask eval/corpora/acquisition "<query>" should answer the acq gold rows the same way the persistent index does. Add it to smoke_e2e.py as a subprocess assertion over a handful of gold rows (start with the four requires_crossref rows that already score 1.000, so a failure is unambiguous), and record the wall clock — the claim being tested is "an ephemeral index beats an ephemeral agent", which is a latency claim as much as a quality one.