How detection works

A KeyHog scan is a pipeline. Files come in one side, findings go out the other. In between, four stages:

files → [chunker] → [prefilter] → [detector match] → [post-process] → findings

Most chunks that fail the cheap prefilter stop there, which keeps full regex evaluation focused on plausible inputs. This is not an unconditional hard drop: a rejected chunk that looks encoded can enter a bounded decode-only recovery pass (recursively decoding up to a max_decode_depth, defaulting to 10), so an encoded secret is not lost merely because its plaintext anchor is absent from the original bytes.

Detection mechanisms

KeyHog does not use one universal test for "secret-like." It composes several mechanisms, and their roles are deliberately different:

Mechanism	Role	Can create a candidate?
Service-anchored detector regex	Matches a vendor or credential-specific shape from detector TOML	Yes
Companion patterns	Finds related fields near a primary match; required = true gates acceptance, while optional companions enrich confidence or verification	Confirms an existing candidate
Structured and multiline extraction	Reassembles assignments and strings that syntax splits across lines or nodes	Yes
Decode-through transforms	Scans supported encoded representations while preserving source attribution. Reverse and Caesar admission uses the active detector TOMLs' decode_transforms prefixes	Yes
Bounded static program recovery	Evaluates recognized side-effect-free JavaScript XOR, explicit-key AES-256-CBC, and CryptoJS/OpenSSL passphrase expressions when every operand is embedded and immutable	Yes
Generic assignment bridge	Extracts values beside credential-role keys when no vendor shape exists	Yes
Shannon entropy	Measures byte-distribution uncertainty for opaque generic values	Yes, on the entropy-discovery path
BPE token efficiency	Rejects language-like values that compress into common subword tokens; eligible candidates use it by default, and detector TOML can tune or disable it	No; precision gate
English bigram discriminator	Distinguishes random alphabetic tokens from pronounceable identifiers, dictionary placeholders, and low-diversity masks inside specific shape and context gates	No; admits or rejects an extracted candidate within those gates
Shape, placeholder, path, and context policy	Rejects examples, references, prose, identifiers, and context-specific noise; entropy owners compile their isolated-token floors and lengths from their detector TOML plausibility table	No; precision gates
Checksums and structural validators	Proves or rejects formats that carry intrinsic validity bits or grammar	Adjusts acceptance/confidence
On-device MoE scoring	Scores ambiguous candidates using local features; never sends content away	Adjusts confidence
Live verification	Optionally asks the owning service whether a surviving credential is active	Adds a verdict after detection

Regex, generic extraction, entropy, and decode-through therefore find different candidate classes. Named regexes and generic assignment extraction create candidates; companions, validators, BPE, English bigram evidence, shape/context policy, and confidence then confirm, reject, or score them. Verification runs only after a candidate survives detection and reporting policy.

Structured extraction accepts balanced Helm actions as render-time syntax. It replaces each action with an inert YAML value while retaining every literal source byte. A Jupyter notebook truncated at end of file is repaired only by closing its open string and container delimiters. Any other syntax error remains a counted coverage gap.

ML participation is detector-owned through [detector.ml]. lift can raise a structural score but cannot veto a match, blend combines model and structural evidence, and authoritative lets the model decide an otherwise ambiguous channel. In the shipped corpus every regex-pattern channel is currently lift; 929 of 934 entropy channels disable ML. The five that do not are all generic owners: generic-api-key, generic-high-entropy-string, generic-keyword-secret, and generic-secret use authoritative, and generic-password uses lift. The model is therefore not a general regex false-positive veto. Its current feature record identifies detector and pattern-versus-entropy channel, but not the exact matched pattern. Pattern-local weak-anchor policy remains outside model conditioning until that provenance is carried end to end. The retrainer refuses mismatched detector/channel records and now requires positive and negative held-out support for every blend or authoritative channel before writing another model.

Before model inference, a cheap probabilistic gate may assign 0.1 confidence to an unaccompanied generic candidate that lacks secret-like randomness. It cannot short-circuit a service-regex match or a candidate backed by a matched companion, including a weakly anchored service pattern: those candidates continue into their detector-owned ML mode. This distinction prevents a corpus-wide randomness shortcut from overriding stronger detector-local evidence.

Static program recovery is a decode mechanism, not arbitrary code execution. KeyHog does not invoke Node.js or evaluate source. It recognizes a bounded grammar for cyclic byte-array XOR and Node-style AES-256-CBC decryption, resolves only literal numeric arrays, Base64-encoded JSON arrays, buffer literals, and empty-separator string joins, then checks binding consistency, UTF-8, AES block shape, and PKCS#7 padding before rescanning the recovered plaintext. Recovered XOR calls and Node AES ciphertext bindings are spliced back into bounded parent context, preserving assignment evidence and absolute source offsets. The CryptoJS dialect additionally requires an exact immutable require("crypto-js") alias, decrypt wrapper, literal passphrase and ciphertext bindings, an OpenSSL Salted__ envelope, and EVP_BytesToKey MD5 derivation. Dynamic values or unsupported syntax produce no derived candidate. The original source still follows the normal detector pipeline. The mechanism is disabled with decode recursion, including under --fast.

BPE is not a replacement name for entropy: it is an independent post-candidate signal. BetterLeaks calls the approach Token Efficiency; KeyHog uses the same broad BPE idea while keeping its own detector schema, thresholds, pipeline, and behavioral evidence.

Terminology matters here: BetterLeaks' public documentation names the feature Token Efficiency and describes BPE tokenization as a natural-language false positive filter; it does not present “BPD” as a separate score. KeyHog names its related mechanism BPE token efficiency, uses cl100k_base, measures UTF-8 bytes per token, and resolves the ceiling per detector. If “BPD” is being used informally to mean a bits/byte or bytes/token density, do not treat it as a third byte-density score. KeyHog also uses a separate English letter-bigram discriminator, but it does not measure bits per byte or bytes per token.

The English bigram discriminator evaluates lowercase ASCII alphabetic runs against an embedded 26 by 26 log-probability model. Digits and symbols end a run. Fewer than six alphabetic characters produce no randomness verdict. A random-token verdict requires a mean score at or below -6.85 and at least three distinct letters. Depending on the calling gate, that evidence can keep an otherwise identifier-shaped random credential or reject a confidently English placeholder. The model does not model arbitrary bytes, numeric keys, hexadecimal or Base64 alphabets as such, or short values. Its current thresholds are scanner-wide constants, not detector TOML fields.

Detector-owned tuning: what each setting changes

Detection policy belongs in the detector TOML whenever the choice is specific to a credential type. Scan-wide CLI/TOML values are operational overrides for controlled comparisons or a corpus-wide policy; they are not a second hidden detector definition. keyhog explain <detector-id> shows the policy declared by that detector TOML and its provenance; keyhog config --effective shows the resolved scan-wide policy.

Practical ownership rule: any numeric value that changes one secret family's recall, precision, shape admission, or scoring must be a named detector-TOML field. If the schema cannot express it, extend the typed schema and its explain/contract surfaces rather than adding a detector-specific literal in scanner code. Only true shared invariants, such as parser safety caps or a model's fixed vocabulary, remain global.

Global structural non-secrets are typed Tier-B data in rules/entropy-universal-rejections.toml. Plain prefixes and explicit prefix-plus-length rules, including the Ag Sealed Secrets ciphertext boundary, apply uniformly before detector plausibility. They are corpus-global structural invariants, not a second source of per-detector tuning.

Detector TOML field	If increased / enabled	If decreased / disabled
entropy_low	Requires more Shannon entropy for keyword-anchored generic values; fewer low-randomness passwords/tokens survive	Admits more values when the assignment key supplies evidence; shape, BPE, context, and confidence gates still apply
entropy_high	Tightens keyword-independent generic admission and raises the partial-confidence tier for entropy fallbacks	Admits more opaque candidates and grants the partial entropy score at a lower value
entropy_very_high	Tightens isolated, anchor-free admission and raises the full-confidence tier for entropy fallbacks	Expands the no-keyword search and grants the full entropy score at a lower value
sensitive_path_entropy_very_high	Raises the keyword-free bar even in sensitive files	Lowers the explicit sensitive-path bar for that detector, improving recall in .env/secret manifests
plausibility.keyword_free_operator_margin	Raises the detector-owned margin composed with the Tier-A entropy threshold	Lowers that explicit margin for the keyword-free role owner; no other detector may declare it
[detector.entropy_fallback]	Changes the emitted synthetic entropy finding identity and semantic class for that detector	Omitting it for an active entropy owner fails compilation; there is no scanner-global compatibility identity
entropy_roles	Claims one or more corpus entry paths: keyword-free, isolated-bare, or unclaimed-keyword	Omitting a role disables that path in a focused custom corpus; no built-in owner or threshold is substituted
decode_transforms.reverse_prefixes	Admits character-reversed candidates that can recover one of the declared plaintext prefixes	Omitting a prefix prevents reverse recovery for that prefix in a focused custom corpus
decode_transforms.caesar_prefixes	Admits only ROT-N shifts that can recover one of the declared plaintext prefixes	Omitting a prefix prevents Caesar recovery for that prefix in a focused custom corpus
[[detector.entropy_shapes]]	charset, optional grouping, diversity requirements, and a lower shape floor admit explicitly structured isolated credentials	Stricter structural requirements or a higher floor narrow the exception; omission is invalid for an active entropy owner
entropy_floor	A higher applicable length-bucket floor suppresses more low-entropy candidates for that detector	A lower floor preserves more human-chosen or structured credentials
plausibility.mixed_alnum_floor	Rejects more identifier-like alphanumeric runs	Preserves more low-randomness mixed-alphanumeric values
plausibility.symbolic_entropy_floor	Raises the minimum entropy for symbol-bearing credential assignments, including the bare auth= bridge	Preserves more anchored symbolic passwords through the same compiled detector policy
plausibility.second_half_entropy_floor	Rejects candidates with a less-random tail	Preserves more credentials whose entropy is front-loaded
plausibility.second_half_min_len	Applies the tail-entropy check to shorter values	Restricts the tail check to longer values
plausibility.unique_chars_min_len	Applies distinct-character requirements to shorter values	Restricts the diversity check to longer values
plausibility.min_unique_chars	Requires more distinct characters once the diversity check applies	Preserves lower-diversity credentials
plausibility.unanchored_hex_max_len	Allows longer unanchored all-hex values before treating them as non-secret key material	Rejects shorter unanchored all-hex values
plausibility.identical_char_max_len	Allows longer single-character repetitions	Rejects shorter single-character repetitions
plausibility.structured_dotted_min_len	Requires a longer isolated structured dotted token	Admits shorter structured dotted tokens after the other gates pass
plausibility.mixed_alnum_min_len	Requires a longer mixed alpha-numeric credential before the carve-out applies	Lets shorter anchored mixed tokens use the detector's mixed floor
plausibility.isolated_mixed_entropy_floor	Raises the floor for isolated contiguous or underscore-delimited mixed tokens	Preserves more low-randomness isolated mixed tokens
plausibility.isolated_symbolic_min_len	Requires a longer isolated symbol-rich credential for the short-candidate exception	Admits shorter symbol-rich candidates; exact declared lower-dash layouts still use their own shape rules
plausibility.isolated_symbolic_min_symbols	Requires more symbol bytes in the isolated symbolic exception	Admits candidates with fewer symbol bytes after the other gates pass
plausibility.isolated_symbolic_requires_non_underscore	Prevents underscore-only mixed tokens from bypassing their mixed entropy floor through the symbolic exception	Allows underscore to satisfy the symbolic exception by itself
plausibility.isolated_colon_left_min_len / isolated_colon_right_min_len	Requires longer sides around an isolated opaque:opaque separator	Admits shorter colon-separated opaque pairs
plausibility.leading_slash_base64_entropy_floor	Raises the floor for unanchored slash-led base64	Preserves more slash-led base64 candidates
plausibility.leading_slash_base64_min_len	Requires a longer unanchored slash-led base64 candidate	Admits shorter candidates after alphabet, padding, entropy, and shape checks pass
plausibility.reject_repeated_blocks	Rejects periodic mask values, including truncated repetitions in the bare auth= bridge	Allows that shape to continue through the remaining detector gates
plausibility.allow_alphabetic_credential	Admits anchored all-letter passwords/tokens after other gates	Requires alphabetic-only values to clear the ordinary entropy path
plausibility.reject_program_identifiers	Rejects pure source-language identifier shapes	Allows pure identifier-shaped values through the remaining gates
plausibility.reject_source_symbol_identifiers	Rejects digit-bearing mixed alphanumeric source-symbol shapes independently of the pure-identifier gate	Allows those mixed values to follow the detector's mixed_alnum_floor and mixed_alnum_min_len policy
plausibility.reject_dash_segmented_alnum	Rejects serial/product-key-like dash groups	Allows dash-segmented alphanumeric values through the remaining gates
entropy_policy_priority	Wins more overlapping generic keyword-policy claims	Yields shared keywords to a more specific detector; unique keywords are unchanged
bpe_max_bytes_per_token	A higher ceiling is looser: fewer compressible/word-like candidates are rejected	A lower ceiling is stricter: more language-like values are rejected, with corresponding recall risk
bpe_enabled = false	Not applicable	Skips token-efficiency rejection for detectors such as human-chosen passwords
decoded_hex_key_material_lengths	Adds only the declared pure-hex widths after transport decoding	Omitted widths remain decoded-digest negatives
canonical_hex_key_material	Generic detectors admit declared lengths only under exact keywords or vendor-prefixed suffixes; regex detectors use length-only entries because their matched pattern is the anchor	Omitted policy, scope, or length remains a digest-shaped negative; there is no service-wide width fallback
min_len / keyword_free_min_len	Longer values are required; short false positives fall, but short real credentials can also fall	Shorter credential shapes become eligible
max_len (entropy-policy owner)	Longer values remain eligible across generic assignment, entropy fallback, and explicit regex envelopes; increase only when the credential contract permits them	Overlength values are rejected whole with value_too_long before entropy or BPE
allowlist_paths, allowlist_values, stopwords	Adds detector-specific path, value-regex, or literal exclusions	Removing an exclusion makes that detector consider the matching path/value again; it does not affect other detectors
pattern required_literals	Routes a prefixless regex only after at least one AST-proven necessary ASCII literal is present. The declaration is the sole owner of non-prefix literal routing	Omitting it leaves the regex in its prefix, keyword-gated, or always-active route; an unsound declaration rejects the detector
public_identifier_assignment_markers	Classifies detector-local assignment-key fragments as public identifiers instead of credentials	Omission disables this suppression for that detector; there is no scanner-global blockchain/network marker list
min_confidence	Raises this detector's reporting floor	Lowers this detector's reporting floor; an operator override can still replace it
detector/pattern weak_anchor	Keeps generic shape/entropy gates active for a whole detector or an individual pattern; requires the owning detector's entropy_high and entropy_floor	Trusts unmarked patterns; use only when those patterns prove the credential shape
structural_password_slot	Applies password-slot placeholder policy to a free-form value captured from a syntactic credential slot	Leaves that detector outside the structural-password family
private_key_block	Makes the detector's enclosing key block suppress less-specific findings nested inside it	Treats the match as an ordinary, non-enclosing finding
[detector.credential_shape]	Declares exact prefix/length/shape constraints that a captured credential must satisfy	Omitting it leaves that detector without an additional credential-shape constraint

Resolution rules

These settings do not all use one generic “last value wins” rule:

• Generic keyword ownership: the highest entropy_policy_priority among detectors claiming the normalized assignment keyword owns entropy and BPE policy. Equal priorities use stable detector identity, independent of corpus order. Custom detector policy keywords join entropy discovery directly; they do not need to be repeated in [scan].secret_keywords.
• Final match resolution: the active compiled plan classifies named, phase-2 generic, entropy, and enclosing private-key findings. The reporting service string and detector-ID length do not change specificity. Unknown finding identities fail checked resolution instead of inheriting embedded or service-name behavior.
• Entropy entry roles: entropy_roles selects the detector that owns each corpus-level entry path. A compiled corpus may have at most one owner for each role. Missing roles remain disabled, and duplicate owners fail scanner construction. Role selection never depends on a detector ID spelling.
• Weak anchors: detector-level weak_anchor = true applies to every pattern, while the same field inside [[detector.patterns]] governs that exact regex. Each such detector owns entropy_high and length-bucketed entropy_floor. Scanner construction rejects an explicit weak anchor without that local policy. KeyHog never guesses this semantic choice from regex text, and min_confidence does not disable it. The compiled hot path uses a primitive detector-indexed floor program.
• BPE ceiling: every active entropy owner declares either bpe_max_bytes_per_token or bpe_enabled = false; omission fails detector validation and scanner construction. An explicitly supplied [scan].entropy_bpe_max_bytes_per_token or --entropy-bpe-max-bytes-per-token replaces every BPE-enabled entropy/generic detector ceiling; the CLI wins over the config file. bpe_enabled = false still disables the gate for that detector.
• Confidence floor: the scan floor defaults to 0.40. A detector TOML min_confidence replaces the scan floor for that detector, and an operator [detector.<id>].min_confidence replaces the detector-declared floor. Under --precision, the resolved global and per-detector floors are clamped to at least 0.85; neither source can weaken the precision preset.
• Entropy policy: every active entropy owner must declare its high, low, very-high, sensitive-path, mixed-alphanumeric, symbolic, tail-entropy, length, isolated-shape, and BPE policy. Scanner construction compiles these into concrete detector-indexed values; a missing field is an error, not a runtime default. Schema defaults remain only for non-owning programmatic detector values that never supply entropy policy. The scan-wide entropy_threshold is deliberately not a blanket replacement for all four bands. On the phase-2 generic bridge it tightens only when it exceeds the owning detector's high band. On the entropy scanner, a value above that high band tightens keyword and isolated candidates; a value below the keyword detector's low band loosens that keyword path, while values between the low and high bands leave its low floor in place. The isolated path keeps its mixed-alphanumeric floor unless the scan threshold exceeds the high band. Named-detector heuristic confidence uses the resolved scan threshold as its partial entropy tier and the scoring margin above it as its full tier; changing the setting can therefore change a named finding's confidence without changing whether its regex matched. These rules preserve the different evidence carried by an assignment key, an isolated opaque token, and an unanchored generic value. The owning entropy_high and entropy_very_high values also define the partial and full heuristic-confidence tiers for emitted entropy fallbacks. Detector ML policy then composes with that heuristic; an authoritative ML mode may replace it, while disabled, lift, and blend modes retain the documented heuristic semantics.
• Sensitive paths: sensitive_path_entropy_very_high is a required detector-local threshold for active entropy owners. Equaling entropy_very_high means no sensitive-path relaxation; a lower declared value is an explicit detector-owned recall choice.
• Credential plausibility: the required detector plausibility block owns its entropy floors, length carve-out, alphabetic admission, and repeated, identifier, and dash-segment rejection choices. There is no production-path fallback for an active entropy owner.
• Synthetic entropy identity: every active entropy owner declares [detector.entropy_fallback] with a semantic class (generic, password, token, or api-key), an entropy-* id, display name, and service. The compiled scanner uses the complete metadata from the active detector corpus for entropy-only findings. Omitting the block is a visible compile error; no scanner-global keyword classifier or compatibility identity can relabel a custom candidate.
• Isolated entropy shapes: generic entropy owners declare one data-driven shape with its character set, entropy floor, optional fixed-width grouping, diversity requirements, and special minimum length. For the shipped lower-alphanumeric app-password policy, candidate length is derived from four groups of four plus three separators; special_min_length controls the short-candidate revisit and must not exceed that derived length. The shape is used for anchorless synthetic entropy recovery; the anchored bluesky-app-password regex remains the source of the named Bluesky finding. A custom corpus without the shape has no isolated exception, rather than inheriting an embedded detector policy.
• Isolated symbolic credentials: the detector's plausibility.isolated_symbolic_min_len, plausibility.isolated_symbolic_min_symbols, and plausibility.isolated_symbolic_requires_non_underscore fields control the shorter symbol-rich exception. Contiguous and underscore-delimited mixed tokens stay under plausibility.isolated_mixed_entropy_floor when the owner requires a non-underscore symbol, and an exact declared lower-dash layout must satisfy its entropy_shapes policy instead of bypassing it as symbolic.

Token efficiency can carry more of the precision burden for a detector whose assignment key or regex already creates the candidate. That is the practical per-detector alternative to making Shannon entropy the decisive signal: use a permissive detector-owned entropy floor appropriate to the credential family, then let its BPE, shape, context, and confidence policy reject word-like noise. It is not equivalent to blindly replacing entropy with one global BPE number, and bpe_enabled alone never creates a candidate. Both configured gates still execute; the current pipeline has no entropy-or-BPE branch.

Detector-owned canonical_hex_key_material is the deliberate exception to the BPE and generic low-diversity/decode-as-data gates. Hexadecimal key bytes tokenize efficiently and use a small alphabet for the same mechanical reasons hexadecimal digests do, so the exact detector-owned contract supplies the discriminator: assignment scope plus length for generic detectors, or matched regex plus length for named detectors. Placeholder, degenerate-repeat, entropy, context, and reporting gates remain active. When ML is enabled, this exact TOML match is structural positive evidence and therefore preserves the detector heuristic floor; the model may raise its score but cannot erase a policy-proven key as if it were an unowned entropy candidate.

Scan-wide settings remain operational controls, but they do not all compose the same way. The operator-layer order and working TOML/CLI examples are in Configuration. Stable per-detector tuning belongs in the owning detector TOML and should be proved with that detector's positive, negative, evasion, backend-parity, and corpus contracts.

Settings, active corpus, and exact identity

KeyHog keeps detector content, resolved scan policy, and corpus provenance separate:

• The reported detector corpus digest binds the normalized corpus schema and the active detector specifications after composition and [detector.<id>] enabled = false removal. A matching disable therefore changes the digest. An unknown disabled ID warns and leaves this digest unchanged.
• The autoroute rules identity also describes the active detector specifications. Operator confidence-floor overrides are composed later so different scan presets can coexist in one calibration cache. It is the canonical corpus identity an execution pack carries, so a scan that compiles the corpus and a scan that hydrates an installed generation of that same corpus read the same calibrated table. Self-test fixtures and declaration order are excluded.
• The autoroute configuration identity binds the resolved scanner and operator policy. It includes the selected fast, deep, or precision preset, scan-wide and per-detector floors, the configured disabled-ID set, detector tuning inputs, worker and pipeline settings, backend/GPU policy, and profiling instrumentation.
• The corpus path and embedded, replace, or overlay label are provenance. They are reported in versioned output, but the path spelling is not detector content. Copying the same normalized corpus to another directory does not create a different content digest. A mode change changes the digest only when it changes the resulting active specifications.

The preset definitions and their override rules are in Configuration. --profile is performance instrumentation, not a named scan-policy profile.

Hardware changes execution, not detection policy. CPU, SIMD/Hyperscan, and GPU routes consume the same resolved detector and configuration identities. Autoroute accepts a candidate only when its canonical detection identities match the reference: chunk membership, detector id/name/service/severity, exact credential, stored hash, companion identity, source, file, line, byte offset, commit, author, date, entropy, confidence, and multiplicity. Mismatch diagnostics name only the differing fields and occurrence counts. They never expose raw values or deterministic value fingerprints.

Built-in suppression, confidence, decode, and scanner postprocessing are already part of those backend results. CLI allowlists and rules, policy floors, cross-source deduplication, verification, and output formatting run after selection. Missing or stale exact evidence is an error. Calibration never relaxes a detector to make a backend look faster.

Two kinds of change exist, and keeping them apart is the whole point of the parity model. A policy change is allowed to change findings. An execution change is not. A finding-set difference across an execution change is a parity failure, which KeyHog treats as a defect rather than a result.

Policy changes: findings may change

Change	Finding-set effect	Routing and calibration effect
Change a preset (--fast, --deep, --precision)	Intended. Each preset resolves a different confidence floor and decode policy.	Configuration identity changes; calibration for the old identity is not reused
Change scan-wide policy, a per-detector floor, or the disabled-ID set	Intended, according to the setting	Configuration identity changes
Change detector TOML, corpus schema, or replacement/overlay membership	Candidates, suppressions, confidence, or final findings may change	Active corpus and rules identity change; recalibration is required
Apply a matching [detector.<id>] enabled = false	That detector stops reporting	Corpus digest changes. An unknown disabled ID warns and leaves the digest unchanged
Change the input	The input can change findings	The route class changes only when the shape of the work changes: byte, chunk, maximum-file, or pattern band, decoder kinds, or the set of source classes

Execution changes: findings must not change

Change	Finding-set effect	Routing and calibration effect
Change CPU, GPU, driver, or accelerator availability	None for the same resolved identities and input. A parity mismatch rejects that route.	Host, device, and runtime identity change; old host evidence is not reusable
Use --backend cpu, simd, gpu-cuda, gpu-metal, or gpu-wgpu	None. Parity-identical by contract.	Diagnostic override. It bypasses autoroute and creates no reusable fastest-correct evidence
Switch between a one-shot process and a ready daemon or watch runtime	None. Runtime lifetime must not change detector policy or canonical matches.	Cold-aware and warm persistent-runtime routes may have different winners
Change --threads, worker, or pipeline settings	None	Configuration identity changes, so calibration is per worker shape
Copy the same normalized corpus to another path	None	Content identity is unchanged; reported source provenance changes

A difference in the first table is a decision you made. A difference in the second table is a bug. Report it with the effective config, detector digest, input identity, backend, host identity, and the complete finding sets from both runs.

Strict Backend Parity

KeyHog exposes three search-backend classes: pure Rust CPU, SIMD/Hyperscan (simd-regex), and GPU/VYRE region presence. Autoroute measures five concrete runtime peers when eligible: scalar CPU, Hyperscan CPU, CUDA, native Metal, and WGPU. Portable builds retain the pure-Rust trigger path without Hyperscan. keyhog calibrate-autoroute rejects any peer whose canonical match identity differs from the reference. It records the first real GPU dispatch plus warm trials: an ordinary process resolves against the cold-aware GPU cost, while a daemon that initialized its engines before readiness resolves against the warm GPU evidence. A missing or invalid decision is not autoroute evidence: KeyHog warns, selects no backend for the affected batch, records incomplete coverage, and prints the exact repair command.

When comparing settings, record the effective config, detector digest, input identity, backend, host/accelerator identity, and complete findings, not only elapsed time or finding count. A faster run with a different result set is a detection change or parity failure, not a routing win.

Stage 1 - chunker

A file becomes one or more chunks. A chunk is {data: str, metadata: {source_type, path, line_offsets, …}}. The chunker:

• Skips obvious binaries via magic-byte sniffing (PDF, PNG, zip, …).
• Skips files matching is_default_excluded_path (node_modules, .min.js, build/, etc.).
• Splits files larger than the 1 MiB window size into overlapping ~1 MiB windows so a single giant log file doesn't blow scratch memory. Each window carries its absolute base byte offset and base line so findings report the real file offset/line, not the per-window one. Cross-window secrets are reassembled in stage 4.
• Decodes UTF-16 BOM files into UTF-8 (PowerShell / .NET configs).

Specialized chunkers run too:

• Git history → one chunk per (commit × file × diff line)
• Docker images → one chunk per layer × file
• Web URLs → one chunk per response body / sourcemap / WASM strings
• S3 buckets → one chunk per object body
• GCS buckets → one chunk per object body
• Azure Blob containers → one chunk per blob body

Stage 2 - prefilter (the cheap pass)

Three gates, in order, each cheaper than the next:

1. Alphabet screen. A 256-bit mask of which bytes the corpus's detectors care about. A chunk with no relevant byte becomes a prefilter miss.

2. Bigram bloom. A 4096-bit bloom filter of 2-byte sequences from detector keyword prefixes. A chunk with no overlapping bigram becomes a prefilter miss. This cheaply recognizes source that carries no relevant anchor vocabulary.

After these screens, ordinary misses stop. Decode-shaped misses instead take the bounded decode-only path described above; transformed plaintext is then attributed back to the original source.

3. Backend trigger pass. The simd-regex backend compiles the detector corpus into Hyperscan databases when the simd feature is present; cpu-fallback uses the pure-Rust trigger path. One pass returns which detector IDs have a candidate match.

   GPU-capable builds add VYRE's resident fused literal-evidence backend. Its single dispatch returns region presence plus detector-derived localization positions; the shared host regexes still decide every finding. There is no universal model-name or byte threshold at which KeyHog silently switches to it. --backend auto requires an exact persisted calibration decision for the current binary, detector/config digest, host/device/driver, workload class, and size bucket. Calibration keeps a GPU route only when its canonical match identities equal the reference and it is the fastest eligible backend for that key.

Stage 3 - detector match

For each pattern-backed detector that the prefilter flagged, the full regex evaluates. The regex is detector.patterns[].regex in that detector's TOML, and its configured capture group becomes the candidate credential. Generic phase-2 detector TOMLs use keyword, length, entropy, token-efficiency, and shape policy for shapeless assignments or isolated opaque values. They may also carry explicit patterns for strongly structured envelopes such as JSON "secret", "token", or "apiKey" fields; both mechanisms remain owned by the same detector TOML instead of a central compatibility detector.

A detector's .toml carries:

• id, name, service, severity, keywords
• zero or more patterns, each with regex + group + optional description (required for service-anchored detectors; optional structured-envelope anchors for phase2-generic)
• optional typed companions; required entries gate acceptance, reinforcing entries add evidence, and forbidden entries suppress
• optional bounded detector_relations with requires, conflicts, or subsumes semantics across findings in the same source, file, and revision
• optional verify block: HTTP method, URL template, auth scheme, success status

Detectors fall into two camps:

• Service-anchored. Regex requires a service-specific keyword (AWS_SECRET_ACCESS_KEY=, stripe.com/v1/, dn_ Deepnote prefix). These have HIGH precision: the keyword itself is positive evidence, not just a hint.

• Generic / entropy discovery (generic-password, entropy-api-key, entropy-token). Triggered by entropy + assignment shape only - password = "...", secret: "...", JSON { "token": "..." }. Lower precision; suppression filters do most of the work.

  Surviving candidates also pass a BPE token-efficiency gate. Shannon entropy asks how evenly bytes are distributed; token efficiency asks how readily a fixed subword vocabulary compresses the value. Dotted API names and prose can have high Shannon entropy but tokenize into a few common pieces, while opaque secrets usually require many short tokens. The mechanisms are complementary, and generic detector TOMLs may own their token-efficiency ceiling through bpe_max_bytes_per_token. Opaque API-key/secret policies use their detector-owned ceiling, falling back to the scan-wide default of 2.2 UTF-8 bytes per token when they do not declare one; password/passphrase policies set bpe_enabled = false because human-chosen credentials may intentionally be word-like. Disabled policies skip tokenizer work entirely rather than using a magic oversized ceiling.

The entropy-generic, entropy-password, entropy-token, and entropy-api-key IDs are output classifications for entropy-discovered findings, not four additional detector TOML files. Their candidate policy is owned by the corresponding phase-2 TOMLs selected from the assignment context: generic-secret, generic-password, generic-keyword-secret, or generic-api-key. Use keyhog explain on those owning detector IDs when tuning entropy, BPE, length, or canonical-key policy.

The split matters for the post-process stage.

Stage 4 - post-process

Even a regex match isn't always a credential. Stage 4 filters:

• Known example fixtures (Stripe docs key, AWS docs key, RFC 7519 specimen JWT).
• Placeholder language - credentials containing YOUR_, INSERT, EXAMPLE, PLACEHOLDER, TODO, FIXME, etc.
• Shape gates.
  • Universal: punctuation_decorated_identifier - credentials starting with --, &, @, !, /, $ (CLI flags, pointers, SQL vars, shell vars, GraphQL refs) or ending in : / ! (UI labels, TypeScript non-null assertions).
  • Generic / entropy only: pure_identifier, word_separated_identifier, scheme_prefixed_uri, url_or_path_segment, contains_uuid_v4_substring. These shapes can be real credentials when paired with a service or protocol anchor, so named detector TOMLs and structural authorization detectors own those cases. A generic token=<uuid> remains an identifier; an Authorization: Bearer <uuid> value is a credential because the Bearer envelope supplies the missing evidence. Public salts and nonces are not generic secrets. A detector for a product whose field is genuinely secret despite that name must own the product syntax explicitly.
• Path-based suppressions - vendored bundles (node_modules/, wp-includes/, bower_components/), CI workflow files (where ${{ secrets.NAME }} references are syntactic, not credentials), i18n translation files, secret-scanner source files (the file IS a scanner; its regex literals shouldn't fire on itself).
• Cross-chunk reassembly. A secret split across window boundaries gets reassembled from the tail of chunk N + the head of chunk N+1.

A finding that survives stage 4 makes it to output.

Semantic source roles and structured parsing

KeyHog analyzes the syntactic role of matched text to distinguish genuine credential assignments from comments, documentation, and mock data.

Each candidate is classified into a SemanticSourceRole:

Semantic source role	Source context	Precision effect
environment-assignment-value	.env files, shell KEY=value lines	Highest confidence for credential assignments
structured-header-value	HTTP request/response headers, YAML/JSON auth blocks	High confidence for credential headers
code-literal	String literals in source code ASTs (.js, .py, .rs, .go, ...)	Standard confidence; subject to identifier and placeholder screens
standalone-token	Bare tokens without key-value assignment anchors	Evaluated through entropy, shape, and BPE token efficiency gates
comment	Single-line and block comments in source files	Downgraded by default unless --scan-comments is enabled
test-fixture	Unit test fixtures, mock data, and test files	Suppressed by default unless --no-suppress-test-fixtures is enabled
documentation	Markdown fenced blocks, docstrings, README files	Suppressed or downgraded according to detector documentation policy
binary-string	Extracted printable strings from compiled binaries	Evaluated under binary strings length and entropy bounds
unattributed	Synthetic findings or callers without semantic indexing	Default neutral baseline
unknown	Files with unrecognized or non-matching structured extensions	Abstained role; candidate evaluates on standard structural evidence

Structured parser scoping and abstention

Structured configuration parsers (dotenv, JSON, YAML, TOML) enforce strict file-extension scoping:

1. Extension-matching paths: Files with recognized extensions (.env, .json, .yaml, .yml, .toml) parse according to their format grammar.
2. Non-matching extensions: Files with non-matching extensions (for example config.unknown or data.txt) abstain to SemanticSourceRole::Unknown. They do not guess format syntax from arbitrary file extensions.
3. Unnamed memory buffers: Unnamed streams (path: None), such as standard input or in-memory chunk buffers, use content-based structural sniffing to identify dotenv or JSON payloads.

Decoded sub-chunk semantic scoping

When an input contains encoded strings (such as Base64, Hexadecimal, or URL-encoded payloads), KeyHog's decode-through engine extracts the decoded bytes into a sub-chunk.

A decoded sub-chunk clears its inherited file path to None during semantic indexing. This ensures that the decoded payload is parsed based on its own syntactic structure rather than inheriting the outer file's extension. For example, a Base64-encoded JSON object inside a .txt file is parsed as structured JSON rather than plain text.

A credential found in both the container bytes and the decoded payload is reported once, at the coordinate in the file you can open. Its evidence is the stronger of the two. A Kubernetes Secret whose base64 data: value decodes to AWS_ACCESS_KEY_ID=... therefore reports likely with the assignment role the decoded text proves, at the offset of the encoded value.

Pattern provenance and secret-safe evidence

Every finding emitted by KeyHog carries a structured provenance record inside its evidence block. This metadata identifies the exact pattern and context that produced the match without disclosing secret material:

{
  "schema_version": 1,
  "detector_digest": "0123456789abcdef",
  "pattern_index": 0,
  "candidate_channel": "pattern",
  "source_role": "environment-assignment-value",
  "context_class": "vendor-pattern"
}

Provenance fields

• schema_version: Version of the provenance schema (currently 1).
• detector_digest: 16-character lowercase hexadecimal hash of the active compiled detector specification.
• pattern_index: 0-indexed ordinal of the matched regex pattern in the detector TOML, or null for entropy-discovered candidates.
• candidate_channel: Pipeline stage that generated the candidate: pattern (regex match), entropy (entropy discovery), companion (companion match), static-recovery (bounded JavaScript XOR/AES evaluation), or unattributed.
• source_role: The SemanticSourceRole where the match was located.
• context_class: Surrounding context category (vendor-pattern, weak-anchor, generic-assignment, standalone-token, or unsupported-context).

Provenance records are deterministic, portable, and safe to share in public CI logs and triage envelopes.

Where the speed comes from

The alphabet screen and bigram bloom reject irrelevant chunks before regex confirmation. Literal triggers narrow the active detector set, and the scanner shares confirmation, suppression, and reporting tails across CPU and GPU backends. Windowing bounds scratch space for large inputs; caches avoid repeated compiler and index work.

End-to-end throughput depends on the detector/config digest, source shape, candidate density, decoding and verification policy, cache state, CPU, GPU, driver, and storage. Use keyhog calibrate-autoroute for routing evidence on the installed host and the repository benchmark harness for reproducible cross-version measurements; do not treat a throughput number from another machine or detector corpus as a routing threshold.

Where the precision comes from

Filter	What it catches
Known example fixtures	Stripe docs key, AWS docs key, RFC 7519 JWT
pure_identifier	getParameter, Benutzername, auth_decoders
word_separated_identifier	s3_secret_access_key (function name)
scheme_prefixed_uri	urn:foo:bar (URI literal, not creds)
url_or_path_segment	/api/v1/users/123 (REST path)
contains_uuid_v4_substring	TOKEN_LIST=636765a9-… (UUID identifier)
punctuation_decorated_identifier	--api-secret, &password, Password:
Vendored-minified-path	node_modules/jquery-3.6.0.min.js
CI workflow path	.github/workflows/ci.yml - ${{ secrets.X }}
i18n translation path	locale/de.po - translated password word

Each filter has a known-FP-cluster it was built to defuse. The Suppressions page enumerates them with examples.

What this looks like for one finding

file.env contains: AWS_SECRET_ACCESS_KEY=ev0BsFtSD7S/4VWYObxiEhME3hJBXeYzR43jgiB1

stage 1 - chunker:        emit chunk{ path: "file.env", data: "AWS_SECRET..." }
stage 2 - alphabet:       PASS (chunk has `=`, alphanumerics from the corpus)
stage 2 - bigram bloom:   PASS (`AW`, `WS`, `_S` are in the bloom)
stage 2 - simd-regex:     MATCH → triggers `aws-secret-access-key` + `generic-password`
stage 3 - regex eval:
  `aws-secret-access-key` detector pattern captures the 40-byte value
    captures `ev0BsFtSD7S/4VWYObxiEhME3hJBXeYzR43jgiB1`
  `generic-password` regex doesn't match (no `_password`/`_pwd` substring)
stage 4 - post-process:
  known-example check: no
  `looks_like_pure_identifier`: false (has digits + /)
  `looks_like_punctuation_decorated_identifier`: false
  → EMIT

That's one finding's life. Multiply by 10⁶ files and the throughput math is why each stage matters.