Testing Strategy

Tools

• Test runners: Vitest for source tests; Node.js node:test for built CLI, installed-package, and documentation tests
• Gate command: pnpm verify (runs documentation checks, package boundaries, lint, typecheck, and every test suite)
• TDD flow: red-green-refactor — write a failing test first, implement the minimal code to pass, then refactor

Three-Package Test Strategy

@cloudburn/rules

Three test layers, all in packages/rules/test/:

Layer	File	What it verifies
1. Export surface	exports.test.ts	awsRules is non-empty, preset inclusion policy holds, azureRules/gcpRules are empty
2. Metadata contract	rule-metadata.test.ts	Catalog fields, IDs, evaluator modes, and declared dataset reads
3. Evaluator behavior	{rule-name}.test.ts	Full finding payloads for both evaluateLive and evaluateStatic, plus negative cases

Keep catalog invariants and semantic metadata policies in rule-metadata.test.ts. It runs each evaluator with empty resource bags and checks that dataset reads match declarations in both directions, rejecting undeclared reads and unused declarations. Lambda memory recommendations also require inventory for SDK evaluation coverage; an explicit assertion pins that dependency pair. SDK export tests verify declared datasets have registered loaders. Evaluator fixtures cover data-dependent reads and complete findings. Avoid copying every rule's name, description, and dependency list into a second declaration-shaped snapshot; keep exact assertions for decisions such as optional evidence or finding precedence.

For static IaC rules, evaluator coverage must include both Terraform-shaped and CloudFormation-shaped resources. A passing test suite for only one source kind is incomplete.

@cloudburn/sdk

Mock at the provider/parser boundary — do not call real AWS APIs or read real files in unit tests.

Boundary	Mock strategy
AWS providers	Mock Resource Explorer catalog helpers and hydrators to return fixture data
Terraform parser	Mock parseTerraform to return { resources, diagnostics } fixtures
Config loader	Mock loadConfig or pass config directly via CloudBurnClient runtime overrides

Test focus areas:

• runStaticScan — registry + static dataset dependency resolution + parser selection + dataset loading + evaluation
• runLiveScan — registry + dataset dependency resolution + Resource Explorer catalog + dataset loading + evaluation
• buildRuleRegistry — mode-aware rule filtering for iac and discovery
• mergeConfig — per-mode merge behavior and runtime override precedence
• CloudBurnClient — facade delegates correctly to engine and provider helpers

Split static AWS provider tests into two layers:

1. Static dataset loader tests per dataset
2. Orchestration tests in loadAwsStaticResources

When adding an IaC rule or dataset, cover Terraform and CloudFormation normalization in provider tests and positive and negative findings in evaluator tests. Keep representative full scans for package wiring, plus dedicated scope, suppression, diagnostics, configuration, and evaluation-evidence regressions. Add a service-specific scanner test when it exercises a distinct integration behavior; repeating every provider and evaluator expectation at the facade adds maintenance without testing another scanner path.

Split live AWS provider tests into three layers:

1. Resource Explorer catalog tests
2. Dataset loader/hydrator tests per service
3. Orchestration tests in discoverAwsResources

Discovery HTTP integration

packages/sdk/test/discovery-http-integration.test.ts runs the public CloudBurnClient.discover() method with the real catalog, hydrators, AWS serialization/deserialization, and rule evaluation. Only the AWS HTTP transport is intercepted. Synthetic JSON and XML responses cover pagination, catalog deduplication, hydration scope, findings, and denied required evidence. Unexpected requests fail the test before reaching the network. Keep focused provider unit tests for exhaustive service behavior.

The same suite verifies ELB inventory reuse through the public pipeline. It counts load-balancer and target-group metadata calls separately from target-health and metric requests, and checks that unsupported types and failed metric series remain visible in unknown coverage alongside valid findings.

packages/sdk/test/discovery-metric-http-integration.test.ts exercises CloudWatch evidence through the real public discovery pipeline. Partial SageMaker invocation data cannot emit an idle finding, and mixed complete/unknown endpoints retain both valid findings and unknown coverage. Provider tests cover status/pagination/retry behavior and Lambda's rolling windows and weighted duration summaries using synthetic AWS responses and fixed clocks.

Reusable evidence tests run through those same HTTP pipelines: unchanged scans, rule selection, credential sessions, view changes, new catalog membership, partial coverage, strict refresh, and independent cancellation. Lambda tests compare actual CloudWatch request windows with cache provenance and verify dependency-version invalidation. The generic cache suite uses real child processes for concurrent refresh, owner crashes, lease expiry, and old-writer fencing, plus synthetic loaders for corruption, invalidation, bounded eviction, and value isolation. No AWS access is needed.

Pipeline tests gate synthetic catalog pages and hydration responses through CloudBurnClient.discover(). They assert that account evidence starts during catalog preparation, a cached catalog type releases before an unrelated miss, and useful provisional results precede final completion. Empty pages with continuation tokens cannot produce a pass. Cancellation after progress stops transport and later events and rejects the final result. Engine integration coverage retains selected optional evidence, finding precedence, and partial-region evaluation semantics.

cloudburn (CLI)

Command tests (*.command.test.ts) mock the SDK boundary to isolate CLI behavior. Their output assertions also cover formatter integration; keep direct formatter tests for distinct layout and serialization edge cases. Shared root option parsing needs representative command coverage, while mode-specific configuration and restrictions stay with their commands. The separate test/e2e/ suite runs the built executable against real Terraform and CloudFormation files, without mocking the SDK or parsers.

Boundary	Mock strategy
CloudBurnClient	Mock .scanStatic(), .discover(), and discovery helper methods
builtInRuleMetadata	Use real built-in metadata unless explicitly testing the empty state

Test focus areas:

• Each command produces correct output for a given ScanResult
• Root and command-local --format resolve to the expected json|table output
• Config-provided mode formats become the default when --format is absent
• --config, --enabled-rules, and --disabled-rules pass the expected runtime overrides to the SDK
• --exit-code sets process.exitCode = 1 when active findings exist
• --fail-on and configured fail-on apply inclusive severity thresholds with CLI precedence
• Suppressed IaC findings remain in output but never set process.exitCode = 1
• Gates without qualifying active findings set process.exitCode = 0
• discover supported-resource-types, discover init, discover status, config, rules list, and estimate all go through the shared formatter system
• table output stays human-readable and json output stays machine-readable
• Runtime errors remain structured JSON on stderr regardless of stdout format

Built CLI template tests

pnpm test:e2e builds the CLI and its workspace dependencies, then starts the executable in isolated temporary directories. Fixtures and reviewed finding expectations live in packages/cloudburn/test/e2e/. These tests cover Terraform, CloudFormation YAML and JSON, positive and negative findings, source scope, source locations, configuration, suppression, parser diagnostics, and exit codes. They use no AWS credentials or account resources.

Keep the fixture expectations independent of implementation output. Normalize only irrelevant result ordering; preserve rule IDs, resource IDs, source locations, and diagnostics. Add representative complete command flows here and keep exhaustive rule permutations in SDK/rules tests.

Installed-package tests

pnpm test:packages builds and packs all three workspace packages, installs the archives into a temporary consumer project, then checks the installed CLI executable and SDK ESM/CommonJS exports with real scans. Nothing is published. Installation can access the public npm registry for runtime dependencies, so this suite is uncached and requires registry connectivity. The temporary consumer uses the repository-pinned pnpm version. The uncached task forwards HTTP_PROXY, HTTPS_PROXY, NO_PROXY (including lowercase forms), and NODE_EXTRA_CA_CERTS through Turbo and into subprocesses for registry connectivity. It does not use AWS credentials or contact AWS.

The built CLI template suite installs a module-resolution guard that rejects AWS SDK and Smithy dependencies, including credential providers. Help/version and static fixture scans must complete with the guard active. Installed ESM and CommonJS SDK consumers use the same guard to check static scans, synchronous region helpers, credential-scope Promise identity, and pre-cancelled discovery methods. A separate installed consumer drives real live chunks in both formats with synthetic HTTP responses, checking signed scoped credentials, findings, owned client disposal, and active cancellation. A module-resolution hook also cancels the first AWS import and checks that late import completion never resolves credentials. These structural checks enforce import independence without machine-specific timing gates.

Runtime and type contracts

Export tests should exercise values imported from the public package entry point. Constructing a typed object and asserting its literal fields does not verify an export or a type contract: Vitest transpiles source tests without checking types, and package typecheck scripts include src only.

Keep compiler-backed consumer checks for type contracts. Existing examples are the rules purchase contract and SDK public contracts, which explicitly run TypeScript. Keep SDK consumer type assertions in that shared test so they use one compiler program during parallel verification.

Fixture privacy

Use synthetic AWS account IDs, ARNs, credentials, and responses in committed fixtures. Do not record live account output or add real credentials, real account identifiers, or live AWS workflows to this test pipeline. The synthetic discovery fixtures use 111111111111; this is example data.

Running Tests

# Single package
pnpm turbo run test --filter @cloudburn/rules

# All suites (including built CLI and installed-package checks)
pnpm test

# Built CLI and real template fixtures
pnpm test:e2e

# Installed archives and public package exports
pnpm test:packages

# One discovery integration file
pnpm --filter @cloudburn/sdk exec vitest run test/discovery-http-integration.test.ts

# Repository knowledge checker and its public CLI tests
pnpm docs:check
pnpm docs:test

# Full verification gate
pnpm verify

docs:test exercises scripts/check-docs.mjs through its process exit status and diagnostics with isolated fixtures. The live docs:check validates required entry points, root and package instruction aliases, repository-contained local links, heading fragments, reference-style definitions and uses, code-example exclusion, canonical page reachability, and the root AGENTS.md line budget.

CI and task caching

CI installs dependencies once in a shared validation job. Pull requests run pnpm verify --affected; pushes to main run the full pnpm verify gate. Both pass --concurrency=2 to Turbo because hosted runners have 4 vCPUs, and running every package's Vitest suite beside the CLI end-to-end tests, lint, and typecheck stretches timer-paced integration tests past their budgets. Documentation checks and package boundaries always run. Turbo selects affected package tasks and shares required builds within the job.

Source tests resolve workspace source directly and can run without dependency builds. The test:inputs transit task propagates upstream source changes into downstream test cache keys without serializing their execution. Built CLI and installed-package suites depend on the CLI build, which depends on SDK/rules builds. Test fixture edits invalidate tests without rebuilding unchanged package output. See the command reference for task dependencies and cache policy.