@intentsolutions/audit-harness

September 13, 2026 · View on GitHub

Part of the Intent Eval Platform — the umbrella grouping the platform's six repos: five converge via a shared Evidence Bundle schema (intent-eval-core, intent-eval-lab, audit-harness, j-rig-skill-binary-eval, intent-rollout-gate), plus intent-eval-dashboard as a satellite consumer (not part of the convergence taxonomy).

Deterministic test-enforcement toolkit. Companion to the audit-tests and implement-tests Claude Code skills — but usable standalone in any repo that wants hash-pinned, escape-scanned, AI-resistant quality gates.

On "AI-resistant", precisely: the hash-pin is a complete guarantee for the files you pin — they cannot change without an explicit, committed re-pin. The escape-scan is a heuristic net over the staged diff: it catches the common escapes (deleted tests, lowered coverage floors, new skip markers, architecture-rule bypasses), not every conceivable dodge. Deliberately not claiming bulletproof.

What it is

A small CLI dispatching 17 released deterministic commands (shell + stdlib-Python scripts) — an 18th, audit-harness migration-notes, is on disk and staged for the next release (see CHANGELOG [Unreleased]):

Command	Purpose
`audit-harness verify`	Verify hash-pinned artifacts haven't changed since `--init`
`audit-harness init`	Pin the current state of engineer-owned policy files
`audit-harness list`	Show pinned files
`audit-harness escape-scan --staged`	Detect AI attempts to lower test thresholds, delete tests, bypass architecture rules
`audit-harness cred-gate`	Provider-credential PASS/FAIL gate — FAIL if a declared secret, provider-key shape, or serialized env leaks into the artifact about to be signed
`audit-harness arch`	Run language-appropriate architecture-rule checker (dependency-cruiser / import-linter / ArchUnit / deptrac / arch-go)
`audit-harness bias`	Count common test-bias patterns
`audit-harness gherkin-lint`	Advisory Gherkin quality check
`audit-harness crap`	CRAP (Complexity × Coverage) scorer — Python, Go, JS/TS, Rust
`audit-harness emit-evidence`	Wrap a gate-result JSON envelope in an in-toto Statement v1 (predicate `gate-result/v1`)
`audit-harness classify`	Read-only repo classifier → an `audit-profile/v1` value (never writes)
`audit-harness conform`	Read-only conformance gate-runner → `gate-result/v1` rows against bundled content-addressed schemas
`audit-harness audit`	Read-only testing-depth gate-runner → coverage presence per pyramid layer + crap-score
`audit-harness scan`	Read-only security/hygiene/skill-quality gate-runner; `--fail-closed` makes applicable OSV measurement release-blocking
`audit-harness fp-rate`	Measure each gate's false-positive / false-negative rate over a labeled corpus
`audit-harness currency`	Advisory poll-freshness report over the per-upstream pin relation
`audit-harness gen-layer-applicability`	Project the canonical audit-profile registry into `layer-applicability.md`

Install

Pick the install flavor that matches your repo's ecosystem — all three publish the same CLI surface.

Node / JS / TS (from npm):

pnpm add -D @intentsolutions/audit-harness
# or: npm install --save-dev @intentsolutions/audit-harness
# or: yarn add --dev @intentsolutions/audit-harness

Python (from PyPI):

pip install intent-audit-harness
# or inside a project venv:
python -m pip install intent-audit-harness

Rust (from crates.io):

cargo install intent-audit-harness

Any other language (Go, Ruby, PHP, Java, .NET, shell, etc.) — vendor the scripts:

curl -sSL https://raw.githubusercontent.com/jeremylongshore/intent-audit-harness/main/install.sh | bash

Quick usage

Pre-commit hook (`.husky/pre-commit`)

#!/usr/bin/env sh
pnpm exec audit-harness escape-scan --staged
pnpm exec audit-harness verify

CI workflow (`.github/workflows/ci.yml`)

  containment:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: pnpm/action-setup@v5
      - uses: actions/setup-node@v6
        with: { node-version: '20', cache: 'pnpm' }
      - run: pnpm install --frozen-lockfile
      - run: pnpm exec audit-harness verify
      - run: pnpm exec audit-harness escape-scan --range origin/main..HEAD

Dependency security in CI

Install the repository-pinned, checksum-verified OSV binary and run the dependency gate in fail-closed mode:

OSV_BIN="$RUNNER_TEMP/osv-bin"
bash node_modules/@intentsolutions/audit-harness/scripts/install-osv-scanner.sh "$OSV_BIN"
export PATH="$OSV_BIN:$PATH"
pnpm exec audit-harness scan --fail-closed --osv-severity-threshold HIGH . \
  > dependency-gate-results.json

A supported lockfile requires a measured scan. Missing/crashed scanners and unparseable results fail. No supported dependency input emits NOT_APPLICABLE only when the repository also declares no dependencies; an unlocked declared graph fails closed. Production or unknown-exposure findings at HIGH or above block, while proven development-only findings remain visible for triage. See Dependency vulnerability gate for the exact contract, evidence fields, and policy exceptions.

Engineer workflow — change a policy threshold

# 1. Edit tests/TESTING.md to change coverage.line from 80 to 75
# 2. Re-init to accept the change
pnpm exec audit-harness init
# 3. Commit the updated manifest alongside the policy change
git add tests/TESTING.md .harness-hash
git commit -m "chore(test): lower coverage floor to 75"

The containment model

The harness enforces this rule: policy changes must be conscious, not silent.

Engineer-owned files (tests/TESTING.md, package test scripts, CI workflows, coverage and mutation configs, acceptance features, and architecture rules) are hashed into a manifest. Any diff that changes their content without a fresh audit-harness init is caught by pre-commit / CI and REFUSED. Ordinary application source is deliberately outside this denominator.

AI agents remain useful (they can read policy, they can implement within constraints). What they can't do is silently weaken the constraints. That's the entire design.

The same walls can close the loop before a push leaves the machine: audit-harness worktree-run --pre-push checks the exact ref being pushed in a disposable git worktree — verify + escape-scan on the push range stay fail-closed, conform + audit contribute advisory gate-result/v1 rows — then removes the worktree. It has no push authority, no LLM stage, and never writes to the repo. Wire it as a lefthook pre-push job (this repo's lefthook.yml is the reference) so CI becomes confirmation instead of discovery.

See audit-tests/references/philosophy.md in the companion skill for the full rationale.

The 7-layer testing taxonomy

This harness sits inside a larger framework:

L7  Acceptance / RTM / Personas / Journeys     ← WHAT are we proving?
L6  E2E / BDD / Visual regression              ← User-level guarantees
L5  Perf / Security (SAST/DAST) / A11y / Chaos ← Non-functional
L4  Integration / Contract / Migration         ← Infrastructure wiring
L3  Unit + Coverage + Mutation + Arch + CRAP   ← Code-level correctness  ← audit-harness lives here
L2  Static analysis / Lint / Types / Secrets   ← Read-only scanning
L1  Git hooks / CI enforcement                 ← The cheapest gate       ← audit-harness enables this

The harness commands serve L1 (escape-scan in pre-commit + CI) and L3 (CRAP, architecture, bias, hash-pin).

Exit codes

Important for CI scripting:

Exit	Command	Meaning
0	any	Clean
1	escape-scan	CHALLENGE — requires engineer-approved comment
2	verify	`HARNESS_TAMPERED` — pinned file changed
2	escape-scan	REFUSE — pipeline halted
3	verify	No manifest (fresh repo, not an error)
1	scan `--fail-closed`	Required dependency measurement failed or policy-blocking OSV findings exist

Language support

Most scripts are language-agnostic (shell + regex). CRAP has per-language backends:

Language	CRAP	Arch	Notes
Python	radon + coverage.py	import-linter	full support
JS/TS	complexity-report + c8	dependency-cruiser	full support
Go	gocyclo + go test -cover	arch-go	full support
Rust	rust-code-analysis + tarpaulin	(custom)	coverage integration pending
Java/Kotlin	—	ArchUnit	via language-native tooling
.NET	—	ArchUnitNET	via language-native tooling
PHP	—	deptrac	via language-native tooling

License

Apache 2.0 — see LICENSE and NOTICE.

Note: versions 0.x shipped under the MIT license. Starting with v1.0.0, the project is licensed under Apache 2.0. Existing 0.x releases on npm remain available under their original MIT terms; new releases (>= 1.0.0) are Apache 2.0.

audit-tests Claude Code skill — diagnostic pipeline that uses this harness
implement-tests Claude Code skill — filesystem-mutating installer that installs this harness as part of L1/L3 setup

Versioning

SemVer. Breaking changes to the CLI surface bump major; new commands bump minor; bug fixes bump patch.

Contributing

This is infrastructure code. Changes need to be conservative. Before opening a PR:

Read audit-tests/references/philosophy.md (in the companion skill) to understand the escape-grammar design
Run bash scripts/escape-scan.sh --staged on your own diff — yes, the harness tests itself
Add test cases if you're adding a new pattern to escape-scan or a new command to the CLI