bench-baselines
July 29, 2026 · View on GitHub
Committed performance reference points for the phase gate described in
BENCHMARKING.md ("Baselines: capture & compare")
and docs/audit/2026-06-architecture-audit.md
(Phase 0.0).
Conventions:
phase-<N>-pre.json— captured on the commit a refactor phase branches from, committed with the phase's first PR. Every PR in the phase runsbench-baseline compareagainst it.phase-<N>-post.json— captured at phase close on the same hardware class and env knobs (FLUREE_BENCH_PROFILE,FLUREE_BENCH_SCALE) as the pre-baseline. All tuples must be within budget vs pre; improvements are banked by tighteningregression-budget.jsonin the closing PR.- Ad-hoc local baselines (validation runs, experiments) should live outside the repo or be cleaned up before merge — only phase reference points belong here.
- Each baseline records a
hostblock (class, name, os, arch, cpu model, cores).host.classnames a comparability set, fromFLUREE_BENCH_HOST_CLASSor the derived{os}-{arch}default.comparerefuses (exit 2) when the baseline's class differs from the comparing host's, since neither wall-clock time nor a tracking allocator's peak survives a machine change;--allow-host-mismatchdowngrades those to non-gating annotations. Per-phase share drift gates either way — it is a ratio within one run, so the machine cancels. - A baseline that also carries a corpus block (sha256, byte length, element
count, input/output syntax, thread count) makes
comparerefuse outright on a corpus change. There is no override: comparing different inputs is never meaningful, and the result would look exactly like a real regression. - Baselines built with
capture --accumulateover ≥ 5 runs carry a median + MAD per scenario, and the gate widens each budget to when that exceeds the declared budget. Prefer this for any baseline meant to gate on shared CI runners.
guardrails-pre.json is the PR-1 (guardrails net) reference — captured at the
merge-base engine, quick profile, tiny+small, schema 1. It predates the host
block, so its runner_class: "local" is read as host.class = "local": an
honest "some developer machine, unspecified". That is why the per-PR compare
still runs with --allow-host-mismatch.
Capture (the --label/--out name the phase reference; capture at the
merge-base commit):
cargo run -p fluree-bench-support --bin bench-baseline -- \
capture --label phase-1-pre --out bench-baselines/phase-1-pre.json