PERF.md

Every entry: fixed hardware, named binary/commit, the profile that led to the change, and the numbers. Companion to benchmark/scale/ (content-hashed sha256 baselines + CI budgets) and the #473 epic's rule: no claim without a rerunnable script. GPU co-residency numbers (recall under 100% MI300X load) live in docs/deployment-amd-mi300x.md.

#630 — embedding quantization: aggressive index compression denoises AND speeds up dense recall

Not a single before/after — a measured ladder. The intuition says compressing the stored vector must trade recall for speed; at 1M the opposite holds on the honest signal. This section records the index-quantization ladder and pins the two orthogonal quantization axes so future rows land in the right column. Companion artifacts: benchmark/embedding-quantization/report.json (the sha256-fingerprinted roll-up) and the per-tier scale runs under benchmark/lambda/results/ (scale1m_exact_ceiling.json, scale1m_2c_on.json, scale1m_default_500.json, scale1m_pure1bit.json). Continuation of #619 (the 1-bit prefilter campaign) — this is #619's ceiling measured against the full-precision cosine it was always compared to in principle but never head-to-head at scale.

The honest signal

1M corpus: 995,562 persisted rows, 10,000 clusters × 100, nomic-768 dim, 2× H100 SXM5, 500 queries, uniform arm, max_scan 50,000. The recall number that means anything here is standalone dense recall. The hybrid keyword arm saturates ~1.0 (r@5 = 1.000 at every tier) because the synthetic corpus tags each cluster with markers BM25 matches exactly — so hybrid recall is NOT the quantization signal and is deliberately excluded from the headline. Dense-only isolates what the vector quantization actually does to ranking.

The two axes (do not conflate them)

1. Index / signature quantization — how the stored vector is compressed for the dense prefilter: full f32 cosine (embedding, 1×) → int4 (emb_sig4, 8×) → 1-bit sign code (emb_sig, 32×). This is the ladder measured below. The tiers are derived from whatever vector is stored, so the ladder isolates this axis on a fixed base model (here nomic-768) across all three rungs.
2. Model-weight quantization — how the embedding itself is generated. The bundled local default is INT8 (all-MiniLM-L6-v2 qint8, 384-dim), used for the offline/default path and the repo's 100K + LongMemEval numbers. The 1M ladder above, though, was measured on the nomic-768 endpoint model, not this bundled default — so the model-weight axis is orthogonal and separately (mostly) pending. FP16/BF16 is measurement_pending (re-embed with the full-precision ONNX export and rerun — cheap at 100K local). NVFP4 is deferred: it needs Blackwell-class hardware, and the interesting question (a larger embedder at NVFP4 vs MiniLM-FP32 at equal memory) is tracked in #629.

Numbers (1M index-quantization ladder, standalone dense, 500 queries)

Tier	compression	bits/dim	dense r@1	r@5	r@10	dense p50	source artifact
full f32 exact cosine	1×	32	0.486	0.684	0.754	650.1 ms	scale1m_exact_ceiling.json
int4 (emb_sig4) coarse	8×	4	0.497	0.722	0.791	395.6 ms	scale1m_2c_on.json
1-bit prefilter + rerank (SHIPPED)	32×	1	0.514	0.726	0.800	194.5 ms	scale1m_default_500.json
pure 1-bit (prefilter, no rerank)	32×	1	0.132	0.312	0.412	184.3 ms	scale1m_pure1bit.json

The headline, counterintuitive and every row measured and content-hashed: more aggressive index quantization gives BOTH higher standalone-dense recall AND lower latency. The 1-bit prefilter (0.726 r@5 @ 194.5 ms) beats full-precision exact cosine (0.684 r@5 @ 650.1 ms) — higher recall at every k, at 3.3× lower latency and 32× less index memory (~96 bytes/vec vs ~3072 at 768-dim; ~96 MB resident at 1M). The 1-bit sign code isn't just cheaper, it denoises: dropping magnitude keeps only the orthant, which discards cross-cluster cosine confusers the full-precision score ranks too highly. int4 sits monotonically between the two on both axes, confirming the trend isn't an artifact of the 1-bit endpoint.

The rerank does the denoising, not the 1-bit ranking (measured)

The shipped 1-bit tier is a prefilter + exact-cosine rerank — the Hamming scan picks the candidate pool, phase-2 cosine reorders it. DenseOpts.rerank / env PERSEUS_VAULT_DENSE_SIG_RERANK=0 (added this cycle; default ON, default path byte-identical) turns the rerank off to measure the prefilter alone. Same 1M corpus, same instance, 500 queries:

	dense r@1	r@5	r@10	p50
1-bit prefilter + rerank (shipped)	0.514	0.726	0.800	194.5 ms
1-bit prefilter, no rerank	0.132	0.312	0.412	184.3 ms

Dropping the rerank saves ~10 ms and halves-and-then-some the recall (0.726 → 0.312 r@5). So the "1-bit beats the f32 ceiling" result above is NOT the 1-bit code being a better ranker — the sign code is a cheap, high-recall candidate filter, and the exact-cosine rerank over that 1-bit-selected pool is what denoises and delivers the 0.726. The rerank is nearly free and does the heavy lifting: keep it on (it is, by default). gate.py locks this in (reranked r@5 must exceed the pure-prefilter r@5 by ≥ 0.2).

Still pending (model-weight axis): the FP16/BF16 row and NVFP4 — #629.

v2.22.0 — release-validation baseline (2026-07-31)

Scope. Fresh internal release validation after the v2.22.0 deployment. Every result below is a measured run, pinned to perseus-vault 2.22.0 revision a9524e9; scale runs use Lambda H100 hardware and LongMemEval is a retrieval-only measurement. These rows do not replace the separately validated public LongMemEval official-CoT QA result.

Exact 10K 2×H100 comparison

10,000 entities; 1,250 clusters × 8; 2,500 queries; Lambda gpu_2x_h100_sxm5.

Metric	Prior canonical	v2.22	Delta
hybrid recall@1	0.900	0.897	-0.003
hybrid recall@5	1.000	0.999	-0.001
hybrid recall@10	1.000	1.000	0.000
hybrid p50 @1	39.6 ms	36.2 ms	-3.4 ms
hybrid p50 @5	39.8 ms	36.8 ms	-3.0 ms
hybrid p50 @10	40.6 ms	38.0 ms	-2.6 ms

Result: no material retrieval regression; modest latency improvement.

Exact 100K 1×H100 comparison

100,000 entities; 1,000 clusters × 100; 2,000 queries; Lambda gpu_1x_h100_sxm5; local nomic-embed-text endpoint.

Metric	Prior canonical	v2.22	Delta
hybrid recall@1	0.785	0.878	+0.093
hybrid recall@5	1.000	1.000	0.000
hybrid recall@10	1.000	1.000	0.000
hybrid p50 @1	94.7 ms	52.9 ms	-41.8 ms
hybrid p50 @5	93.1 ms	53.0 ms	-40.1 ms
hybrid p50 @10	95.6 ms	53.9 ms	-41.7 ms
dense recall@1	0.680	0.711	+0.031
dense recall@5	0.859	0.875	+0.016
dense recall@10	0.899	0.910	+0.011

v2.22 seed throughput was 202.4 entities/s; embedding throughput was 47.0 entities/s, with 99,526 stored vectors. This clears the R2 acceptance targets: hybrid recall@5 = 1.0 and hybrid p50 <100 ms at 100K.

Shared-memory, live serving, and LongMemEval retrieval

• Shared-memory scale: 16 isolated concurrent MCP agents: shared retrieval 1.000, private-memory isolation 1.000, recall p95 9.236 ms against a 250 ms budget — PASS. This is a single-host isolation/concurrency result, not a multi-node claim.
• Live flywheel: after deployment of the full bundled-embedding v2.22 image, the combined live run verdict was PROMOTE: lifecycle 10/10, AAR 7/7, Cycle 3 8/8, Cycle 4 5/5, adversarial 14/14, chaos 7/7, and context-to-receipt 6/6.
• LongMemEval _s, retrieval-only: 500 questions / 23,867 sessions; explicit bundled-ONNX embedding population before query. Auto and hybrid both measured recall@1 84.6%, recall@5 97.4%, recall@10 99.2%, MRR 0.903; dense was 76.8% / 93.2% / 97.2% / 0.842 and FTS5 was 4.2% / 23.4% / 42.2% / 0.126. This is not a QA-accuracy claim.

Artifacts: benchmark-runs/v2.22-release-validation-2026-07-31.md in the release evidence archive; the live flywheel artifact is /opt/data/benchmark-runs/v222-live-flywheel-20260730T234705Z/.

#530 — GPU contention + agent economics: recall unaffected at 100% accelerator load

Not an optimization — a measured property of the architecture, recorded here with the same rules (named hardware, rerunnable script). Perseus Vault's memory layer is host-CPU-resident and uses 0 bytes of GPU HBM; the claim "recall steals no inference cycles from a co-located model" was measured on rented hardware on 2026-07-09. Harness: benchmark/contention/ (live_bench.py for the real-serving rows, burn_bench.py for the synthetic row). Every row is tagged measured (timed live) or projection (derived from published specs) — there are no projections below.

Hardware A: AMD Instinct MI300X 192 GB (RunPod, $2.19/GPU-hr retail), vLLM 0.19.1 + ROCm 7.13, host AMD EPYC 9474F, serving Qwen2.5-72B-Instruct bf16. Hardware B (cross-vendor baseline): 2× NVIDIA H100 SXM 80 GB (Lambda, $8.38/hr, NVLink), same model, same vLLM version, same bench parameters.

Recall under accelerator load (100K-entity store, host CPU)

Condition	Recall p50	Δ vs idle	data_source
GPU idle	18.7 ms	—	measured
GPU serving 72B under sustained load	18.8 ms	±0.6% median (6 idle-vs-serving runs, range −0.4% to +1.1%)	measured
GPU 100% util, synthetic FP16 matmul (97.4 TFLOPS sustained)	—	+0.6%	measured

Recall on the host CPU is flat whether the accelerator is idle, serving a 72B model flat-out, or pinned by a synthetic compute burn: the memory layer and the accelerator do not contend. This section doubles as a regression guard — if the recall path ever grows GPU or lock contention, these deltas move.

Agent economics (measured throughput × measured rental price)

Metric	1× MI300X	2× H100 SXM (best boot)	2× A100 80GB (eager@0.97)	8× A100 40GB	data_source
GPUs (count the cards before comparing)	1	2	2	8	measured
Holds Qwen2.5-72B bf16	one card	1 card cannot load it; 2 required	2 required	needs the pooled 320 GB	measured
Concurrent 8K-token agents (vLLM KV-cache ceiling)	15.3	5.0 (eager-only, 97% util — the only config that boots)	6.37	57.9 (7.2/card)	measured
GPU $/agent-hour	$0.143 ($2.19 ÷ 15.3)	$1.68 ($8.38 ÷ 5.0)	$0.436 ($2.78 ÷ 6.37)	$0.275 ($15.92 ÷ 57.9)	measured
Sustained output tok/s	658	—	746	2,304	measured
$/1M output tokens	$0.92 (untuned bf16)	—	~$1.04	$1.92	measured

Measured-vs-measured, the MI300X's /agent-hour advantage is **11.7×** vs the 2× H100. A100 rows measured 2026-07-11/12 (Lambda 8× A100 40GB SXM4 \15.92/hr; RunPod 2× A100 80GB SXM4 $2.78/hr at the identical eager@0.97 config as the H100 — same model, same vLLM 0.19.1). Read the 8× row with its GPU count: its $0.275/agent-hr comes from eight cards whose 320 GB is heavily overprovisioned for a ~136 GB model, inflating KV headroom — per card the MI300X leads 15.3 vs 7.2 agents and wins 1.9× on $/agent-hour. Scope, stated plainly: untuned out-of-the-box vLLM, bf16 weights, no FP8, no speculative decoding — treat the MI300X numbers as a floor. Deliberately NOT recorded here: single-process serving-throughput floors from earlier runs (wrong serving shape). Reproduce: serve any OpenAI-compatible endpoint, then `python benchmark/contention/live_bench.py --gpu-price <$/hr> --vllm-log ; synthetic row via python benchmark/contention/burn_bench.py`.

#511 — hybrid recall: the "fusion-machinery overhead" was the sparse arm's join-per-match hydration

Hardware: same box as #476/#507 (AMD64 16-core, Windows 11). A/B on an IDENTICAL loaded 100K store (seeded once by benchmark/scale/run.py --keep-db on post-#543 main, kept), 100 queries/mode over MCP stdio with the scale harness's exact query shape and limit, before = main + the timing instrumentation only, after = this change. Every number below is measured.

Where the time went (stage attribution, new PERSEUS_VAULT_RECALL_TIMING=1)

Windows has no cheap flamegraph story for a release binary driven over MCP stdio, so this change adds the permanent alternative: opt-in per-stage timing on the recall path (PERSEUS_VAULT_RECALL_TIMING=1, one line per query to stderr, zero cost when off). 30 timed hybrid queries at 100K, p50 per stage:

Stage	Before	After	data_source
embed (query vector)	6.1 ms	6.2 ms	measured
dense arm	23.1 ms	25.4 ms (concurrent)	measured
sparse arm (BM25)	247.4 ms	60.3 ms (concurrent)	measured
graph expand	0.02 ms	0.02 ms	measured
RRF fuse	0.10 ms	0.09 ms	measured
usefulness boost	0.45 ms	0.33 ms	measured
metadata filter + truncate	0.01 ms	0.01 ms	measured
total	277.0 ms	67.2 ms	measured

The issue's suspects, settled by the profile: RRF candidate over-fetch + hydration, query expansion, graph expand's link following, and the post-RRF weighting passes are all ≤ 0.5 ms combined. 88% of hybrid's cost was the sparse arm: its single SQL joined EVERY FTS match to entities and evaluated the full 24-column row — multi-KB body_json overflow chains included — before ORDER BY rank LIMIT k could discard it. A broad term matches most of the corpus, so that was ~100K record reads per hybrid recall: the #476/#507 disease, third instance, hiding in the FTS5→entities join.

The fix (two changes, exactness-preserving)

1. Two-phase BM25 (fts5_bm25_search): phase 1 ranks rowid + bm25() entirely inside the FTS index — the entities table is never touched; phase 2 hydrates and metadata-filters only a bounded pool (3× the arm's limit, floor 128) in rank order. If filters eat the whole pool it widens once (4096), then falls back to the exact single-query plan — the answer is never silently truncated, and both plans share one predicate builder so they cannot drift. Result sets are identical (see recall gates below).
2. Concurrent arms: the dense and sparse arms are independent read-only queries on separate pooled connections (Database is already shared as Arc<Database> across server threads, #210) — hybrid now pays max(dense, sparse) instead of dense + sparse.

Numbers (100K entities, identical store, 100 queries/mode)

Mode	Before p50/p99	After p50/p99	Δ p50
hybrid	295.5 / 337.0 ms	80.9 / 93.8 ms	3.7×
dense	25.5 / 31.0 ms	25.1 / 30.9 ms	unchanged
fts5	16.5 / 19.9 ms	17.0 / 20.7 ms	unchanged (noise)

Hybrid vs the sum of its arms: 7.0× before (295.5 vs ~42 ms) → 1.9× after (80.9 vs ~42 ms) — inside the issue's ≤2× acceptance target. The scale-gate hybrid p99 budget tightens 1000 → 250 ms so the before-state (p99 ≥ 282 ms on this box) can never silently return.

Recall-quality gates (before AND after, same stores/seeds)

• benchmark/recall/ (deterministic, bundled ONNX): byte-identical — same report signature (d78c7240…), hybrid recall@5 95.8 both sides. recall@5 regression: 0 pt.
• benchmark/longmemeval/ retrieval-only (no LLM, no API key), first 100 instances, paired: recall@5 93.0 → 93.0, recall@10 99.0 → 99.0. recall@1/@3 wobble ±3 pt run-to-run in this regime, but the baseline binary differs from ITSELF on 53/100 per-question top-10s across reruns (async auto-embed-on-write race in the harness's ingest-then-immediately-query loop) — the wobble is harness self-noise, not a ranking change; the deterministic harness above is the ranking gate.

Residual (next target)

The remaining hybrid-over-dense cost IS the sparse arm now: BM25 must score every FTS match to rank them (~60 ms for ~100K matched rows), an engine-level O(matches) floor for broad terms, concurrent with (and larger than) the dense arm. Shrinking it means capping or pre-pruning the match set — a recall trade, not free machinery — so it stays out of scope here.

#507 — dense recall: covering index for the phase-0 signature scan (v18)

Hardware: same box as #476. A/B on the IDENTICAL loaded 100K store (kept from the scale harness), 50 queries/mode, before = the #476-merged binary, after = this change; the after binary's first open runs the v18 migration.

Where the time went

dense_search phase 0 — the "cheap" sign-bit prefilter — and the embedded-row count both predicate on embedding IS NOT NULL. embedding/emb_sig are late ALTER columns stored AFTER body_json in each record, so evaluating the predicate (and reading emb_sig) walked every row's multi-KB body overflow chain: ~900MB of page reads per dense query at 100K. embedding_coverage(), consulted per recall to pick the default mode, paid the same walk.

The fix (v18)

idx_entities_dense_sig ON entities(archived, id, emb_sig) WHERE emb_sig IS NOT NULL — every column the phase-0 queries touch is an index column, so they plan as USING COVERING INDEX (~60B/row; zero record reads), pinned by a plan-text regression test. The queries re-key on emb_sig IS NOT NULL, made exact by the v18 invariant "embedded ⟺ signed": the migration backfills emb_sig from every stored embedding (pure sign-bit recompute, no model), and writers already set/clear both columns together. Two variants that do NOT work, for the record: the embedding IS NOT NULL spelling never covers (residual predicate seeks the table), and an expression index only covers on SQLite ≥ 3.5x — newer than the bundled engine.

Numbers (100K entities, identical store)

Mode	Before p50/p99	After p50/p99	Δ p50
dense	360.3 / 390.9 ms	24.9 / 29.3 ms	14.5×
hybrid	593.9 / 620.7 ms	308.4 / 325.3 ms	1.9×
fts5	14.6 / 17.9 ms	17.0 / 19.4 ms	unchanged (noise)

Residual (next target)

Hybrid ≫ dense + fts5 (308 vs ~42 ms): roughly 265ms lives in the hybrid-only machinery (RRF fusion candidate over-fetch and hydration, query expansion, graph expand's per-candidate link following) — filed as its own issue with this A/B as the baseline.

#476 — write path: signature-driven near-duplicate scan (v17)

Hardware: AMD64 16-core (AMD Family 26), Windows 11 · benchmark/scale/run.py, MCP stdio, one persistent process, seeded corpus, ~40–120-word bodies (~9KB stored).

Where the time went

The per-write near-duplicate scan walked every same-category entity row: each candidate's multi-KB body_json was hydrated (overflow pages included) and re-hashed with body_hash64 as the signature freshness guard — even though the #392 signature machinery could already decide verdicts without the body. Cost per write: O(N·body_size) — ~90MB read+hashed per insert at 10K rows, ~900MB at 100K. Attribution was confirmed empirically before the fix: removing the embedding stack entirely (lite build) still showed the 15× first-to-last-10% degradation, and the opt-in FTS prefilter (#228) measured slower than the scan it pruned (a 64-term OR MATCH per write).

The fix (v17, exactness-preserving)

1. dedup_signatures gains scope columns + a (category, workspace_hash, tg_count) index; a one-time migration backfills every active row's signature ("every active row has a signature" becomes an invariant).
2. The scan walks signatures only — small fixed-size rows, SQL-band-pruned by the lossless trigram-count ratio bound (J ≥ t ⟹ min(a,b)/max(a,b) ≥ t), then the existing lossless count/histogram prunes + exact merge verdict.
3. Freshness moves to verify-on-hit: only a candidate whose signature says "dup" gets its body fetched and re-checked (hash + scope + archived), with self-healing repair. Never a false positive; the deliberate trade is that a row rewritten behind the engine's back can be missed (one extra stored row) until it self-heals — the old guard taxed every write for everyone to cover that rare case. The lossy FTS prefilter is retired outright.

Numbers

Metric	Before (2.19.0)	After (this change)	Δ
10K load, sustained	141/s	554/s	3.9×
10K load, first→last 10%	1107 → 68/s (16×)	1197 → 349/s (3.4×)	5.1× at the tail
100K load, sustained	7/s (~4.0h wall)	39/s (43min wall)	5.6×
100K load, first→last 10%	117 → 3/s	483 → 18/s	6× at the tail
100K fts5 recall p50/p99	16.5 / 181.7 ms	16.1 / 21.7 ms	p99 spikes gone¹
100K as_of p99	0.32 ms	0.26 ms	unchanged path
100K cold start	70.2 ms	71.7 ms	unchanged path

¹ The baseline's fts5 p99 outliers were dedup I/O pressure from the write phase's page-cache churn; with bodies out of the scan they disappear.

Measurement note: the AFTER runs shared the machine with an API-paced LongMemEval harness (bursty local ingest); the BEFORE baselines ran clean. The improvement figures are therefore lower bounds.

Verdict-correctness is pinned by the randomized differential property test (find_near_duplicate_signature_path_matches_exhaustive_scan_property) against the verbatim pre-#392 exhaustive reference, plus contract tests for the verify-on-hit guard (no false positives, self-heal) and raw-row visibility. Full suite: 383 passed.

Residual (next targets)

The remaining 10K tail decay (1197 → 349/s) is embed-on-write CPU (the lite build measured a flat ~35% embed tax) and FTS5/WAL growth — see #507 (dense recall brute-force scan, the read-side sibling) and the scale-gate budgets that lock today's numbers in.