keyhog benchmarks

July 30, 2026 · View on GitHub

A reproducible, multi-corpus, multi-config evaluation harness. One importable Python package (bench/) replaces the scattered scripts that used to live under tools/secretbench/scoring/: every run produces one RunResult JSON on a single schema, so one scorer, one report generator, and one test suite serve every scanner × config × corpus × host combination.

The top-level ../README.md benchmark panels are generated between  markers. make report owns the competitor tables. make readme-matrix owns the KeyHog accuracy, configuration, daemon, CPU, reader, storage, corpus-size, and concurrent-partition panels. Never edit generated rows by hand.

Quick start

make mirror     # generate the 15k synthetic SecretBench-mirror corpus (off-git, neutral layout)
make keyhog     # build the ci-lean candidate required by backend parity tests
make bench      # run every scanner on the mirror -> results/<host>/
make ioc-recovery-corpus # generate the deterministic P0-P12 recovery corpus
make ioc-recovery # compare all four keyhog presets on exact recovery
make report     # render reports/ + inject the leaderboard into ../README.md
make readme-scaling # measure and inject CPU, reader, storage, size, and partition panels
make readme-matrix # run both README benchmark generators and inject every panel
make readme-matrix-check # verify both snapshots, reports, and README generated bytes
make analyze    # print top FP/FN examples for detector tuning
make test       # pytest the package (scorer truth, loaders, injection idempotence)
make bloom KEYHOG_BIN=/absolute/path/to/keyhog # pinned CredData Bloom parity evidence
make targets    # execute aspirational recall/competitor target specs (expected worklist)

The benchmark KeyHog binary is resolved from $KEYHOG_BIN, else the cargo target-dir in $CARGO_TARGET_DIR, else ~/.cargo/config.toml, else the repo target dir, else keyhog on PATH. Build the benchmark candidate first: make keyhog or cargo build --release -p keyhog --bin keyhog --no-default-features --features ci-lean. The ci-lean timing seam requires an independent authorization value and exists only for deterministic backend parity tests. Do not use this candidate as production release evidence. The benchmark needs create and delete access beside that binary to protect its execution snapshot. For a managed read-only install, copy the verified artifact into a private writable runtime directory and set $KEYHOG_BIN to that copy. The default test/release gate excludes tests marked target_spec; those are executable product targets, not claims that the current release already meets. Run them explicitly with make targets.

Bloom prefilter evidence

make bloom KEYHOG_BIN=/absolute/path/to/keyhog rebuilds the canonical samsung-creddata-fx-record-spans-v1 fixture from pinned CredData commit f1de3f85dbdf42bf7b3467c0d273a4dfe44d56ee, measures the production Bloom gate, and runs byte-identical enabled and diagnostic-bypass scans. It writes results/bloom-creddata-fx-record-spans-v1.json and reports/bloom-creddata-fx-record-spans-v1.md only after at least one real input is rejected and canonical finding identities match exactly.

The receipt accounts for all declared records as measured or unavailable, categorizes every unavailable reason, and binds each finding identity to detector ID, source path, line, byte span, and a credential SHA-256. Credential plaintext is never written. A missing source file, revision mismatch, zero rejection, or finding mismatch fails the command instead of publishing partial evidence.

What it measures

Detection: TP / FP / FN → precision / recall / F1, overall and per-category. General secret corpora use the SecretBench containment rule. Recovery records require the scanner to surface the exact recovered plaintext, so an encoded value or a larger containing string earns no recovery credit.
Speed: wall time + throughput (MB/s) + peak RSS per run. In-process rows use /usr/bin/time -v. Daemon rows time the warm client request and read the owned server's VmHWM, since the server owns scanner memory.
Host: OS / kernel / CPU / cores / RAM / GPU + VRAM, captured per run so desktop / santhserver / Windows-ThinkPad / macOS results aggregate into one matrix.
Workload identity: every new result hashes the complete, sorted scan tree plus the canonical answer-key records (including scoring mode and source coordinates). The SHA-256 identifies the exact bytes and labels, not just a corpus nickname or fixture count.
Config: keyhog's backend × cache × daemon × mode axes; each competitor carries its own knob (kingfisher confidence, etc.) in the same config_id.
Resolved scan manifest: KeyHog benchmark artifacts persist the exact preset, effective detection values, and compatible overrides from the report's resolved_scan object. A mode label alone is never treated as proof that two runs used the same detection policy.
Bounded static recovery: current KeyHog rows carry the versioned static_recovery object with exact supported, unsupported, erroneous, and per-reason rejection counts from the normal scan artifact. The schema checks disposition/reason conservation and static-recovery.md renders exact zeroes. Selected bench-v3 artifacts predate this telemetry and are labeled legacy; the report never substitutes invented zeroes.
Execution route: each KeyHog result records whether execution was in-process or daemon-served. A daemon result also records the owned server PID and exactly two served requests (one warmup, one timed scan).
Detector provenance: every KeyHog run scans a private immutable snapshot and records the SHA-256 of its exact detector TOML filenames and bytes. The gate rejects results whose digest does not match the workspace corpus.
Executable provenance: every warmup and measured KeyHog command runs the same held-inode byte snapshot beside the resolved runtime. Linux launches use the inherited descriptor, so path replacement cannot change executed bytes. Windows holds the snapshot handle through measurement. Darwin fails closed until equivalent current_exe and loader semantics are proven. Results record the snapshot SHA-256 and version. The gate requires an exact match with the current candidate binary.
Source freshness: current-source evidence requires a clean tracked Git tree and an exact recorded HEAD commit. KeyHog validates the snapshot before scanning, and result-only gates check workspace identity before scoring.

Two fairness rules (baked into every corpus)

Both were proven against the live keyhog binary, not assumed:

The answer key is excluded from the scan tree. A scanner pointed at a dir containing manifest.jsonl (every labeled secret in plaintext) "finds" those secrets; betterleaks fires 9392 spurious matches on the mirror manifest, kingfisher 7581. Layout keeps the manifest a sibling of the scan tree: <home>/manifest.jsonl beside <home>/corpus/<shards>; scanners only ever see corpus/.
The scan dir has a neutral name (corpus, never fixtures/test). keyhog applies a path-context test-fixture confidence penalty under "fixtures"-shaped paths that --no-suppress-test-fixtures does not override; the same 15k files scored 1880 findings under fixtures/ vs 2484 under a neutral name.

On the fair mirror keyhog ranks #1, well ahead of every competitor on F1 at near-top precision. The exact figures are not hand-written here; they drift with the binary and would go stale - read them from the generated leaderboard in the top-level ../README.md, regenerated by make report. Always bench the freshly-built release binary (see Reproducibility below); a stale keyhog on PATH invalidates the run, and the cross-device harness rejects preinstalled PATH binaries before recording a row.

Leakage-safe dataset splits

Any corpus used to train, calibrate, and report a sealed score must pass the split leakage guard before measurement:

make -C benchmarks leakage-audit \
  LEAKAGE_MANIFEST=corpora/splits.jsonl \
  LEAKAGE_CONTENT_ROOT=corpora/source \
  LEAKAGE_RECEIPT=reports/leakage-receipt.json

The JSONL manifest has an exact schema. Each row supplies record_id, split, corpus_id, corpus_sha256, repository_family, credential_family, secret, content_path, credential_start, and credential_end. Splits are train, calibration, or sealed-test. Paths are relative POSIX paths under the content root. Credential offsets are Unicode character offsets. repository_family must identify the common ancestry root across forks and mirrors. credential_family must identify the normalized credential lineage, not only its provider label.

The guard fails if repository ancestry, normalized credential lineage, normalized credential value, or an exact or near-duplicate redacted context crosses a split. Context fingerprints exclude the credential and use 200 characters on each side. The token checks retain the published SecretBench leakage thresholds: unique-token Jaccard at 0.80 and multiset Jaccard at 0.70. Byte 5-grams and normalized structure 4-grams add independent 0.90 checks for small edits and identifier or literal rewrites. These settings follow the methodology in From Data Leak to Secret Misses.

The receipt binds the manifest, pinned corpus digests, redacted source content, split assignments, grouping results, and policy. It reports raw and deduplicated sample counts without credential, repository, family, or context bytes. Detected leakage writes a failure receipt and exits with status 2. Malformed provenance fails before scoring. Input is capped at 100,000 records, 8 MiB per content file, 512 MiB across unique content files, and 5,000,000 candidate comparisons. Repeated paths reuse one decoded source buffer.

Layout

benchmarks/
  bench/                  importable package
    schema.py             RunResult + nested records (the common contract)
    hardware.py           host capture
    score.py              corpus-owned exact or overlap attribution scorer
    corpus_integrity.py   manifest and scan-tree digest verification
    leakage_guard.py      provenance and near-duplicate split isolation
    generator_checksums.py shared checksum-valid synthetic-token primitives
    corpora/              mirror · homefield · creddata · ioc-recovery · perf adapters
    scanners/             keyhog (+config matrix) · betterleaks · kingfisher · trufflehog · titus · noseyparker
    runner.py             one (scanner,config,corpus) measurement -> RunResult
    leaderboard.py        the matrix: run many scanners/configs, write results/<host>/, rank
    report.py             results -> markdown + idempotent README injection
    analyze.py            top false-negative / false-positive examples from the same scorer
    gate.py               regression + differential gate (keyhog must lead; CI forcing function)
    tests/                pytest
  generators/             corpus generators (not git-ignored)
    mirror/               synthetic SecretBench-shape generator (generate.py + providers/negatives/wrappers)
    homefield/            competitor home-turf harvesters (harvest_betterleaks.py · harvest_kingfisher.py)
    ioc_recovery/         deterministic P0-P12 JavaScript recovery generator
  corpora/                generated data (git-ignored; reproducible through Make targets)
  results/<host>/         one RunResult JSON per run (git-ignored; regenerable)
  reports/                generated markdown: leaderboard · perf · recall-gap · category-recall · static-recovery
  baselines/              committed known-good scoreboard anchors (regression history)

Corpora

name	what	labels	get it
`mirror`	15k synthetic SecretBench-shape (3k positives / 12k negatives)	yes	`make mirror`
`homefield-betterleaks`	betterleaks' own `tps`/`fps` rule examples	yes	harvested
`homefield-kingfisher`	kingfisher's own rule examples	yes	harvested
`creddata`	Samsung/CredData (~11k files, pinned commit)	yes (T=pos, F/X=neg)	`make creddata`
`ioc-recovery`	336 sources across P0-P12 JavaScript concealment phases (4,368 fixtures)	yes, exact plaintext	`make ioc-recovery-corpus`
`kernel`	Linux kernel tree	no (perf only)	set `KEYHOG_BENCH_KERNEL`
`daemon-file`	one regular file for owned-daemon latency	no (perf only)	set `KEYHOG_BENCH_DAEMON_FILE`

Benchmarked tools and datasets are credited with their licenses in THIRD_PARTY_NOTICES.md. Competitor corpora and CredData are gitignored / fetched locally; keyhog redistributes none of their data.

make creddata checks out the pinned dataset revision, repairs dirty nested source checkouts, rebuilds an existing data/ tree, and validates every metadata fixture before reporting readiness. It uses eight repair workers by default; set CREDDATA_JOBS=<N> to match network and disk capacity.

Use an isolated Python environment when the downloader dependency is absent:

python3 -m venv .venv-creddata
.venv-creddata/bin/python -m pip install pybase62==1.0.0
make creddata PY=.venv-creddata/bin/python

The adapter checks this dependency before it removes or rebuilds data/. It does not install into the system Python environment.

During acquisition, a repair marker makes the corpus unavailable to every reader. After successful validation, the adapter removes the marker and the downloader's transient tmp/ repository clones. A failed acquisition or completeness check keeps both for diagnosis. A partial nonempty data/ tree is reported as unavailable to scorers instead of failing pytest collection or masquerading as a complete corpus. Metadata inventory tools can still declare individual missing inputs explicitly.

Exact secret recovery benchmark

ioc-recovery is an official corpus in the same adapter, runner, scorer, result, and reporting system as every other KeyHog benchmark. It adapts the P0-P12 progression from Benchmarking Large Language Models for IoC Recovery under Adversarial Code Obfuscation and Encryption to deterministic synthetic, checksum-valid GitHub-token-shaped values:

phase	transformation
P0	plaintext baseline
P1-P4	Base64, identifier changes, dead code, and structural changes
P5-P6	XOR and AES-256-CBC with embedded recovery material
P7-P12	XOR or AES combined with the simple, dead-code, and structural transforms

The paper evaluates 336 programs across these 13 phases. KeyHog's generator therefore emits 4,368 JavaScript fixtures by default. The authors publish 13 demonstration files at commit 91d45377cf482c1de6c36a0d33744665976a19b6. That repository does not contain the 336-program evaluation corpus. KeyHog records the repository and commit as methodology provenance, but generates its own deterministic credential fixtures. It does not claim byte identity with the paper's evaluation data and uses no third-party dataset bytes. Generated corpus receipts pin the v1 paper PDF URL, byte length, and SHA-256.

Every value is synthetic and deterministic. P5-P12 embed the key and recovery logic in the program, matching the program-analysis task rather than creating a brute-force benchmark. Generation requires Node.js for its standard crypto implementation of AES-256-CBC. The generator verifies each encrypted value round-trips before publishing the corpus. Focused tests cover every phase with deterministic samples; the expensive hosted deep row covers the complete 4,368-fixture product.

Run the mode comparison with:

make -C benchmarks ioc-recovery

The result directory contains ordinary RunResult JSON for full, fast, deep, and precision. Phase categories (recovery/p00-* through recovery/p12-*) make the exact capability boundary visible instead of collapsing it into one aggregate score.

The Make target holds the backend at deterministic SIMD so differences isolate scan policy. A GPU host can measure the backend and mode cross-product without changing the corpus or scorer:

cd benchmarks
python -m bench leaderboard --corpus ioc-recovery --scanners keyhog \
  --matrix backend,mode --out results-ioc-recovery-backends

Each row records host hardware and the full config identity. Explicit GPU rows use --require-gpu, so an unavailable or failed accelerator is reported as an unavailable result instead of being timed as a silent CPU fallback.

make -C benchmarks targets includes the executable deep-mode target: all 4,368 expected plaintexts recovered exactly, with no blind P0-P12 phase. It is marked target_spec because it is an explicit, expensive capability gate. The older local v0.5.41 recovery files are unavailable/stale rows and are not release evidence. There is currently no checked-in headline recovery result; only a fresh hosted bundle that passes the contract below may establish one.

AgentRE Linux recovery slice

The AgentRE adapter pins the upstream commit, license, task map, 13 Linux C sources, ground truths, build script, and scorer by SHA-256. Source material and binaries stay under the gitignored benchmarks/corpora/ tree.

make -C benchmarks agentre-corpus

The build uses the pinned Linux amd64 GCC image manifest with networking disabled, a read-only source mount, no capabilities, bounded memory and PIDs, and a non-writable container root. It compiles each source with the upstream flags but never runs a sample. Publication succeeds only when all 13 ELF files match their pinned digests and the compiler-bound receipt. The published files are read-only and non-executable. Running the module without --ensure validates an existing corpus:

cd benchmarks
python -m bench.agentre_build

The local scorer reproduces the pinned standard and bonus rubrics, including partial decoded-C2 credit, set overlap, nested fields, hallucination penalties, and rounding. Differential tests compare it with the validated upstream scorer. Analyzers use the explicit benchmark boundary instead of the generic secret corpus runner:

from bench.corpora.agentre import AgentREBenchmark

benchmark = AgentREBenchmark()
tasks = benchmark.tasks()
report = benchmark.score_analyzer_outputs(outputs_by_task_id)

tasks() returns all 13 validated binary paths and their canonical identities. Scoring requires one mapping output for every task, validates all 149 expected fields, revalidates the sealed binaries, and attaches the score-contract receipt before returning the report. The report also separates decoded-C2 recovery from negative absence. The Linux slice has 11 tasks with a decoded C2 and two tasks whose correct value is null. Positive results are counted as exact, host-partial, or missed. Negative results are counted as absent or spurious. The official aggregate remains unchanged, but a recovery-rate claim uses only the 11 positive tasks. The official bonus weights sum to 0.95 although its summary declares a 1.0 bonus maximum. KeyHog exposes both the declared 2.0 total and attainable 1.95 total in a separate score-contract receipt. It does not silently normalize the upstream result. No 100% claim is valid until the complete runner also proves field-level score, backend parity, and fail-visible analyzer coverage.

Daemon measurements

Daemon performance uses a separate unlabeled, single-file corpus because the production daemon accepts stdin or one regular file and redacts credentials in the client report. Raw matches cross the private user-only socket, but the CLI forbids plaintext rendering on this route. It does not accept directory trees or benchmark-only plaintext scoring flags. Linux measurements launch the immutable executable snapshot as a foreground server on a private socket, verify the peer PID with SO_PEERCRED, require request counters 0 -> 1 -> 2, and stop and reap that exact process. Active requests must return to zero after each client. The default user daemon is never contacted.

Only explicit simd, cpu, gpu-cuda, and gpu-wgpu backends are eligible. auto lacks a persisted selected-backend receipt. Cache, fast, deep, precision, and confidence axes are also unavailable because those policies are not bound into daemon startup. Unsupported combinations produce unavailable rows with the exact reason. Explicit CPU and SIMD daemon backends disable all GPU runtime work. Explicit GPU daemon backends require GPU preflight and hard-fail on runtime degradation.

cd benchmarks
KEYHOG_BENCH_DAEMON_FILE=/path/to/representative-file \
  python -m bench leaderboard --corpus daemon-file --scanners keyhog \
  --matrix backend,daemon --out results-daemon

The published README matrix uses a generated deterministic 8 MiB file (sha256:afafbe7b6487fd62866f510e7c281a9e7bfeaa8dc585d7b0478c92ee6c4f5ef5) and the 15,000-fixture mirror corpus:

make readme-matrix

This command requires every explicit CPU, Hyperscan, CUDA, WGPU, policy, cache, daemon, worker, reader, storage, corpus-size, and partition row. It writes two provenance-bound snapshots:

reports/readme-matrix.json owns accuracy, route, policy, cache, and daemon evidence. It also renders accuracy-matrix.md, configuration-matrix.md, and daemon-matrix.md.
reports/readme-scaling.json owns worker, reader, corpus-size, storage, and concurrent-partition evidence. It injects five detailed tables directly from measured trials.

The scaling generator creates three exact finding-free workloads: 256 files at 32 KiB each, 1,024 files at 64 KiB each, and 2,048 files at 128 KiB each. It uses powers of two plus the host's exact effective CPU limit. Concurrent partition runs divide that CPU limit across one, two, and four processes when the host has enough cores. Worker rows use a warm client page cache. Reader, size, storage, and partition rows request clean-page eviction with posix_fadvise when the platform supports it. Every row records the actual policy.

By default, storage measurements compare the workspace filesystem with the local temporary filesystem when they are distinct. Supply explicit storage classes when you need named mounted volumes:

make readme-scaling \
  README_SCALING_STORAGE_ARGS='--storage workspace=/data/keyhog-workspace --storage local-nvme=/mnt/nvme/keyhog-scaling'

Each storage root receives byte-identical corpora. The snapshot stores the filesystem type, device ID, and SHA-256 for every copy. It never stores an absolute path.

A missing route, failed process, non-success scan status, finding in the synthetic corpus, stale binary, mixed identity, changed workload byte, or omitted source-state classification fails capture. Release evidence requires a clean tracked source tree.

For a local comparison that intentionally measures uncommitted changes, label both generators rather than presenting the result as release evidence:

KEYHOG_BENCH_ALLOW_DIRTY=1 \
  make readme-matrix \
    README_MATRIX_SOURCE_STATE=developer-dirty \
    README_SCALING_SOURCE_STATE=developer-dirty

Run make readme-matrix-check in documentation gates to reject stale generated bytes from either snapshot. The nightly hosted CPU workflow also uploads readme-scaling.json and readme-scaling.md for the exact four-core runner.

Tiers

quick: every scanner at its default config (the README leaderboard).
perf: keyhog's in-process backend × cache × mode matrix on a tree corpus, plus the constrained daemon matrix above on daemon-file.

Hosted CPU release evidence

bench-nightly and differential-bench use ubuntu-24.04 four-effective-core profiles in cpu-gates/. Admission requires exactly four logical and affinity-visible CPUs. A cgroup-v2 quota must be observed as exactly four cores or explicitly max; missing, unreadable, malformed, and unobservable/v1 quota state is unknown and fails closed. Runner image/version and runtime supply identity are separate receipts rather than assumptions implied by the label.

The nightly policy requires current rows for mirror, CredData, and recovery fast/full/deep/precision. Context capture derives exact per-category positive denominators from the authenticated workload snapshot. Every scorer row must expose those truth categories and conserve TP/FN against its overall counts. A category absent from positive truth is admissible only with zero TP/FN, and its false positives remain subject to overall FP conservation. Each recovery mode additionally applies the policy's independent P00-P12 floors. The fast preset disables decoding. Its policy requires recall for plaintext, identifier-obfuscation, dead-code, and structural-obfuscation rows, while its Base64, XOR, and AES floors are zero. The full, deep, and precision presets keep positive floors across every P00-P12 category.

The job generates/acquires source corpora outside the absent snapshot destination, then runs python -m bench.hosted_cpu_gate context. It copies each workload, records source/snapshot hashes, and the workflow makes the snapshot root-owned on a read-only bind mount before measurement. A rejected write probe is embedded into the final context; a pre/post workload check remains a second guard. Scanner rows consume only the mounted snapshot roots and bind the exact final context bytes.

The context also binds the Git source, release binary and detector corpus, reviewed policy, repository/workflow/ref/SHA/run/attempt/job, effective CPU allocation, acquisition revision, category denominators, and workload counts/ bytes/digests. Accelerator inventory is explicitly best-effort NVIDIA observation and is not a physical no-accelerator attestation. CPU evidence instead relies on the ci-lean build, explicit CPU/SIMD config, hidden GPU feature environment, and enforced in-process CPU route.

Only cpu or simd in-process rows with cache and daemon disabled are admissible. The benchmark validation mode and the KeyHog preset are separate axes. The full resolved scan manifest (preset, effective policy and overrides) is policy-pinned rather than inferred from either the mode or filename. Authenticated positive category denominators must match the scorer truth categories. False-positive-only categories remain visible and must conserve the overall false-positive count. Each row has an explicit minimum recall, maximum wall time in milliseconds, minimum throughput in MiB/s recomputed from bound bytes/wall time, and maximum peak RSS in KiB. Raw JSON booleans, integers and finite numbers are validated before schema decoding; impossible confusion-matrix totals and exits outside KeyHog's 0/1/10 success domain fail closed.

The committed thresholds remain unmeasured-release-requirements: they are acceptance limits, not measured performance or calibration claims. A reviewed hosted run now pins the CredData and mirror workload identities, every resolved scan manifest, runner image version, and runtime libhs SHA-256. These values are exact identities, not wildcards. A changed image, library, workload, manifest, or policy digest keeps the gate red until you inspect fresh evidence and deliberately update the pin. Do not publish a performance headline from these acceptance limits.

The final command is deliberately result-only. It requires current UTC and run identity from the workflow invocation rather than trusting self-authored receipt fields, including a literal reviewed policy digest:

python -m bench.hosted_cpu_gate gate \
  --policy cpu-gates/github-ubuntu-24.04-4core-nightly.json \
  --policy-sha256 '<reviewed-policy-sha256>' \
  --context hosted-cpu-context.json --unicode-parity unicode-parity.json \
  --trusted-now '<current-utc>' --repository "$GITHUB_REPOSITORY" \
  --workflow-ref "$GITHUB_WORKFLOW_REF" --workflow-sha "$GITHUB_WORKFLOW_SHA" \
  --run-id "$GITHUB_RUN_ID" --run-attempt "$GITHUB_RUN_ATTEMPT" \
  --job "$GITHUB_JOB" --root .

CPU/SIMD parity includes Unicode detector examples explicitly. The only accepted Rust receipt line is exactly backend parity: <N> detector examples; CPU == SIMD on all ASCII inputs; 0 unicode-input divergences, emitted once after the Rust test rejects every mismatch. The reviewed policy pins the exact example count, parity source, and source-plus-detector vector digest. bench.unicode_parity also hashes the exact test executable before/after execution and binds it to the release binary, detector set, policy, context and GitHub run. Positive, missing, duplicate, count-shrunk and malformed output fails closed.

Gate (CI forcing function)

python -m bench gate is the single regression + differential gate (it replaced the retired tools/diff_bench runner). It runs, or with --results <dir>, consumes, a leaderboard, selects the canonical row per scanner with the same newest-wins logic as the README table, and exits non-zero unless keyhog both:

leads every available competitor on F1 strictly (--no-beat-competitors to skip), and
clears the floors you assert: --min-f1 / --min-precision / --min-recall, and/or --baseline <RunResult> + --epsilon (regression vs a committed anchor in baselines/).

Exit codes: 0 pass · 1 violation · 2 undecidable (keyhog produced no usable result). differential-bench builds and measures its own checked-out commit before running both the F1 competitor gate and the independent hosted CPU evidence policy. bench-nightly separately measures and publishes its own mirror, CredData, and recovery bundle; neither workflow consumes the other's results.

Continuous-improvement loop

make -C benchmarks loop runs the whole improvement cycle in one command:

pytest (scorer self-tests) -> ensure corpus -> leaderboard (detection + speed
for every scanner) -> calibrate (per-detector min_confidence floor overlay)
-> render reports/ -> gate (differential + regression vs baselines/)

It is the local mirror of the scheduled lanes that keep keyhog improving across vectors without manual babysitting:

detection / differential: differential-bench (nightly): bench gate fails red if a competitor overtakes keyhog on F1 or keyhog regresses past baselines/canonical.toml; its separate hosted CPU policy also enforces the mirror recall/resource receipt.
hosted CPU recall + resources: bench-nightly (nightly): fail-closed mirror, CredData, and P0-P12 mode rows on a low-core no-GPU runner.
strict recall under evasion: runners-nightly (the Rust strict matrix).
test depth: ci (all_tests, property, e2e on every push).

loop deliberately does not --inject the README: the published tables are regenerated only by make report on a machine with every competitor installed, so a partial-scanner run can never degrade the committed leaderboard. The calibration.toml overlay it emits is the actionable "what to tune next" signal.

Cross-device

cross_device.sh DEVICE=<ssh-alias> rsyncs the current tree to a device, installs keyhog via its per-OS build (Linux: Hyperscan SIMD; macOS: --features portable, the system-lib-free vyre CPU path), generates the corpus on the device's local disk, runs the leaderboard there, and pulls the per-host RunResult into results-cross-device/<device>/. The remote driver always installs from the synced tree and never treats a keyhog found on PATH as current-code evidence. Results stay out of results/ so a remote host's row can never shadow the canonical README numbers (the README report's canonical_leaderboard picks newest-per-scanner across everything it loads). Compare every host with python -m bench.cross_compare; the committed snapshot is reports/cross-device.md. Windows is POSIX-incompatible with this script; drive the ThinkPad via PowerShell.

Reproducibility

Scoring passes --no-gpu for deterministic simd-* configs. Generic local results still record physical GPU inventory, but hosted CPU release evidence requires both the explicit no-GPU environment receipt and an empty runner GPU inventory; auto and GPU configs are inadmissible. Exact GPU rows elsewhere use --require-gpu and fail instead of timing another backend. CredData is pinned to an exact commit and every hosted workload additionally carries a scan-tree + answer-key SHA-256. Overlap scoring remains bit-identical to the retired SecretBench scorer; exact recovery is deliberately stricter. Focused contract tests cover stale/wrong identities, recall and resource limits, missing modes/phases, GPU fallback, and Unicode CPU/SIMD divergence.