Testing Guide

June 19, 2026 · View on GitHub

Testing ensures AgentOps skills and the ao CLI work correctly across changes. This guide covers what tests exist, how to run them, and how to write new ones.

Test Types

Type	Location	Description
Unit (Go)	`cli/*/_test.go`	Go unit tests for CLI internals
Integration	`tests/integration/`	Shell scripts testing CLI commands, skill invocation, hook chains
Smoke	`tests/smoke-test.sh`	Quick sanity checks that the plugin loads correctly
Windows smoke	`tests/windows/test-windows-smoke.ps1`	Native Windows smoke for PowerShell installers, Codex plugin staging, `ao doctor` hints, and focused Windows-sensitive Go tests
Contract	`tests/scripts/*.bats`, `scripts/check-contract-compatibility.sh`	Schema and contract validation
BATS	`tests/scripts/*.bats`	Unit tests for repo shell scripts using the BATS framework
E2E	`tests/e2e/`	Full pipeline proof runs
Skill	`tests/skills/`, `skills/*/scripts/validate.sh`	Skill structure, frontmatter, and behavior validation
Doc	`tests/docs/`	Documentation link and count validation

Test Tiers

The master runner tests/run-all.sh organizes tests into tiers by speed and dependency requirements:

Tier	Name	Requires	What it covers
1	Static Validation	Nothing (runs offline)	Manifest schemas, JSON validity, GOALS.yaml, doc links, skill counts, token budgets, artifact consistency
2	Smoke Tests	Claude CLI	Plugin load test, `smoke-test.sh`, Codex integration
3	Functional Tests	Claude CLI, Go	Explicit skill requests, natural language triggering, Claude Code unit tests, release smoke tests, integration tests

Run a specific tier:

./tests/run-all.sh              # Tier 1 only (fast, no CLI needed)
./tests/run-all.sh --tier=2     # Tier 1 + 2
./tests/run-all.sh --tier=3     # Tier 1 + 2 + 3
./tests/run-all.sh --all        # All tiers

Running Tests Locally

Scenario	Command	Approx. Time
Quick static validation	`./tests/run-all.sh`	~10s
Full test suite	`./tests/run-all.sh --all`	2-5 min
Go unit tests	`cd cli && make test`	~15s
Fast local cockpit gate	`ao gate check --fast`	~30-90s
Install push-to-main cockpit gate	`bash scripts/install-pre-push-gate.sh`	~1s
Go build + vet + changed-scope race	`scripts/validate-go-fast.sh`	~20s
Changed-scope derived artifact repair	`bash scripts/regen-changed-scope.sh --check --scope head`	~1-30s
AgentOps contract canaries	`scripts/test-agentops-contract-canaries.sh`	~2-5m
AgentOps eval advisory corpus	`scripts/eval-agentops.sh --fast`	~5-10m
BATS script tests	`bats tests/scripts/*.bats`	~10s
Skill validation	`tests/skills/run-all.sh`	~30s
Skill integrity (heal)	`bash skills/heal-skill/scripts/heal.sh --strict`	~15s
Doc validation	`./tests/docs/validate-doc-release.sh`	~10s
Contract compatibility	`./scripts/check-contract-compatibility.sh`	~10s
Release-wide local sweep	`scripts/ci-local-release.sh`	5-10 min
Native Windows smoke	`powershell -ExecutionPolicy Bypass -File .\tests\windows\test-windows-smoke.ps1`	~1-3 min

Use scripts/regen-changed-scope.sh for ordinary slice work. It inspects the changed files and runs only the relevant generator/checker pair: Codex hashes for skill edits, context-map generation for skill frontmatter changes, contract index checks for docs contracts, registry generation for registry sources, and CLI command surfaces for cli/cmd/ao command changes.

scripts/regen-all.sh is the release-wide sweep. Use release-wide regen-all for release preparation, command deletion/rename cleanup, large skill-prune waves, or when the changed-scope script explicitly says it has no localized repair.

Examples:

bash scripts/regen-changed-scope.sh --check --scope head
bash scripts/regen-changed-scope.sh --scope worktree
bash scripts/regen-changed-scope.sh --list --file skills/discovery/SKILL.md
bash scripts/regen-all.sh --check  # release-wide / full sweep

AgentOps Contract Canaries

The official blocking contract canary list lives at tests/canaries/agentops-core-official.txt. These are deterministic product contract tests, not model evals; they use ao eval run only as the execution and artifact substrate. Run them before changing selected canary suites, core contracts, or the local/backstop gate that enforces them:

scripts/test-agentops-contract-canaries.sh

The broader public corpus still lives under evals/agentops-core/ and remains advisory while brittle exact-string checks and local-baseline ratchets are hardened. Run it when changing eval contracts, ao eval, or the advisory canary suite:

scripts/eval-agentops.sh --fast

Remote compute has a deterministic canary at evals/agentops-core/remote-compute-contracts.json and a focused shell suite at tests/scripts/test-remote-compute-contracts.sh. Run it while iterating on remote-compute RC slices:

scripts/eval-agentops.sh --suite evals/agentops-core/remote-compute-contracts.json

The contract canary runner writes artifacts under .agents/tests/contract-canaries/ and does not compare local baselines. The advisory eval runner writes artifacts under .agents/evals/runs/. If a promoted baseline exists under .agents/evals/baselines/, the advisory runner compares the new run against it and fails on regressions. In --fast mode it also writes coverage.json and fails when required domains, score dimensions, or deterministic runtimes are missing. Missing baselines are warnings, not failures, so new canaries can land before their ratchet is promoted.

Baseline promotion is explicit and requires a rationale:

scripts/eval-agentops.sh --fast \
  --promote-baseline \
  --promoted-by "$USER" \
  --rationale "Intentional behavior improvement after review"

CI runs agentops-contract-canaries as a blocking test job. CI also runs agentops-eval-advisory, which is intentionally non-blocking while the broad corpus, baselines, and variance policy are stabilized. Live Claude/Codex runtime checks remain advisory unless a suite explicitly opts into a deterministic mock or fixture.

Writing New Tests

Local Hooking

Install the shared Git pre-push cockpit gate explicitly:

bash scripts/install-pre-push-gate.sh

That chains the shared .git/hooks/pre-push path to scripts/hooks/pre-push.local, preserving Git's pre-push stdin even when an earlier legacy hook segment consumes it. On pushes to main, the hook runs the full race suite, then serializes the mutable ao gate check --fast and pawl proof path. The tracked .githooks/ directory is legacy dev-hook plumbing; do not treat it as the live release authority.

Where to put tests

Test type	Directory
Go unit tests	Next to the source file in `cli/` (e.g., `cli/internal/goals/measure_test.go`)
Script tests (BATS)	`tests/scripts/`
Skill validation	`skills/<name>/scripts/validate.sh`
Integration tests	`tests/integration/test-<name>.sh`
Native Windows smoke	`tests/windows/`
E2E proof runs	`tests/e2e/`
Doc validation	`tests/docs/`
Goal validation	`tests/goals/`
Lint allowlists	`tests/lint/`

Naming conventions

Go test files: name after the source file they test (e.g., measure.go -> measure_test.go).
No cov*_test.go naming. Test files must not use the cov* prefix convention.
BATS files: <descriptive-name>.bats in the appropriate tests/ subdirectory.
Shell integration tests: test-<name>.sh.

Assertion rules

No coverage-padding tests. Every test must assert behavioral correctness, not just presence. Tests that use trivial != "" or != nil assertions solely to inflate coverage metrics are banned.
If a function's coverage is low, write a real test that validates behavior or accept the metric gap.

Go Testing Rules

Coverage floor

The Go coverage floor is 84%. CI enforces this. Run coverage locally:

cd cli && go test -coverprofile=coverage.out ./...
go tool cover -func=coverage.out | tail -1

Command / test pairing

Each CLI command file in cli/cmd/ao/ should have a corresponding *_test.go file. Tests should exercise:

Flag parsing and defaults
JSON output mode (--json)
Error paths (missing args, invalid input)
Behavioral correctness of the command's core logic

Assertion density

Tests must make meaningful assertions about output content, exit codes, and side effects. A test that only checks err == nil without validating the result is insufficient.

Script Testing (BATS)

Repo shell scripts (scripts/*.sh) are tested using the BATS framework. Test files live in tests/scripts/.

AgentOps 3.0 is hookless — it ships no hook scripts, so there are no hook BATS tests. The BATS suite covers the repo's shell scripts.

Writing a BATS test

#!/usr/bin/env bats

setup() {
    REPO_ROOT="$(cd "$(dirname "$BATS_TEST_FILENAME")/../.." && pwd)"
    export REPO_ROOT
}

@test "my-script: does the expected thing" {
    run bash "$REPO_ROOT/scripts/my-script.sh" --json
    [ "$status" -eq 0 ]
    echo "$output" | jq -e '.someField == "expected"'
}

@test "my-script: exits non-zero on bad input" {
    run bash "$REPO_ROOT/scripts/my-script.sh" --bogus
    [ "$status" -ne 0 ]
}

Running BATS tests

# All script tests
bats tests/scripts/*.bats

# Single file
bats tests/scripts/agent-output-validate.bats

# Verbose output
bats --verbose-run tests/scripts/*.bats

Skill Testing

Per-skill validation

Each skill can have a scripts/validate.sh that checks skill-specific invariants. The runner tests/skills/run-all.sh iterates over all skills in skills/ and:

Verifies SKILL.md exists with YAML frontmatter and a name: field.
Checks declared dependencies exist as sibling skill directories.
Runs scripts/validate.sh if present.
Runs lint checks (lint-skills.sh), Claude feature coverage, and alias collision detection.

Running skill tests

# Full skill validation suite
tests/skills/run-all.sh

# Skill integrity check (references, orphan files, structure)
bash skills/heal-skill/scripts/heal.sh --strict

heal.sh --strict

The heal script validates that every file in skills/<name>/references/ is linked from the skill's SKILL.md. Missing links break CI.

Quarantine Policy

Tests requiring external services (API calls, network access, live Claude/Codex sessions) that cannot be mocked are placed in tests/_quarantine/.

Promotion path: To move a quarantined test into the main suite:

Replace external calls with mocks or API stubs so the test runs headlessly.
Move the test file to the appropriate tests/ subdirectory.
Verify it passes in CI without network access.

Quarantined tests are excluded from the default run-all.sh tiers and CI. Standalone runtime smoke tests that do not require live runtimes belong in tests/skills/ or tests/scripts/ and are expected to run in CI.

Test Directory Map

Directory	Purpose
`tests/skills/`	Skill validation scripts plus standalone runtime smoke tests (Claude Code, Codex, OpenCode)
`tests/spec-consistency/`	Spec consistency gates across manifests and docs
`tests/goals/`	Goal validation and measurement (`GOALS.yaml` / `GOALS.md`)
`tests/lint/`	Lint allowlists and code style checks
`tests/explicit-skill-requests/`	Tests for explicit skill trigger patterns
`tests/cli/`	CLI flag consistency and behavior tests
`tests/windows/`	Native Windows installer, plugin staging, and focused CLI portability smoke tests
`tests/e2e/`	End-to-end proof runs (full pipeline)
`tests/docs/`	Documentation validation (links, skill counts, goal counts)
`tests/scripts/`	BATS tests for repo scripts (`scripts/*.sh`)
`tests/integration/`	Integration tests (CLI commands, skill invocation, hook chains)
`tests/fixtures/`	Shared test fixtures and sample data
`tests/lib/`	Shared test helpers and color utilities
`tests/_quarantine/`	Quarantined tests requiring external services