Verification Environment Design

August 4, 2026 · View on GitHub

Use this reference when an agent can change code but cannot safely or cheaply exercise the real environment. The goal is not to maximize mocking. The goal is to construct the smallest environment that preserves the behavior needed by the verification claim, then state exactly what that environment does not prove.

A fast fake can be the right environment for request shaping and the wrong environment for database isolation, browser behavior, compiler output, concurrency, or vendor compatibility. Select the substitute only after naming the claim and the dependency semantics that decide it.

flowchart LR
  C["Verification claim"] --> D["Discover existing seams"]
  D --> B["Classify each boundary"]
  B --> F["Choose minimum fidelity"]
  F --> T["Run tracer case"]
  T --> O["Challenge the oracle"]
  O --> E["pass | fail |<br/>unobserved | blocked"]
  E --> R["Calibrate and record gaps"]

Ownership Boundary

This reference owns:

discovery of repository-owned test-environment seams;
claim-driven selection among fakes, virtual services, emulators, ephemeral real dependencies, and authorized sandbox checks;
the environment contract and lifecycle shape;
oracle independence, negative controls, and mock-drift calibration; and
environment safety, isolation, cleanup, and portability.

It does not own case discovery, the regression skeleton, probe verdicts, or diff-to-case scoping; use Agent Verify Loop. It also does not own general diagnostic instrumentation (Observability), recovery evidence (Recovery Evidence), or production authorization.

Load When

A required service, device, account, dataset, kernel feature, or vendor sandbox is unavailable to the agent.
A test suite passes with extensive mocks, but it is unclear which real behaviors remain proven.
Local verification requires manual environment setup, shared credentials, fixed ports, or persistent state.
A container or emulator exists, but nobody has checked whether it preserves the semantics relevant to the change.
The agent must add a reproducible environment before it can close an Agent Verify Loop exercise-and-judge path.

Start With the Claim

Write the claim before selecting a tool:

Given <pinned starting state>, when <real subject action> runs,
observe <evidence> at <boundary>, and decide <expected invariant>,
under <platform / time / authority constraints>.

This turns “we need a mock database” into a decidable question:

If the claim is “the repository calls save() with these fields,” an in-process spy may be enough.
If the claim is “the migration, constraint, transaction, or query works on PostgreSQL,” PostgreSQL semantics are part of the claim; use an ephemeral real instance or an explicitly compatible emulator.
If the claim is “checkout succeeds against the payment provider,” a local stub can prove request construction and failure handling, but provider acceptance remains unobserved until an authorized sandbox or contract verification runs.

Keep the code under change real. Replace a collaborator only at an explicit boundary, and only when its omitted behavior is not part of the claim.

Discover Before Constructing

Do not begin by adding a new mock framework or Dockerfile. Inventory the repository in this order:

Operating instructions and CI: AGENTS.md, contribution guides, workflow files, build scripts, package scripts, Make targets, and test matrices. Extract the commands the project already treats as executable.
Pinned environment inputs: lockfiles, runtime/version files, container tags or digests, generated schemas, compiler flags, browser/device matrices, environment-variable schemas, locale/timezone/seed settings.
Existing seams: fixtures, fake factories, __mocks__, test builders, in-memory transports, local servers, service virtualizers, emulators, Testcontainers, Compose services, recorded snapshots, contract tests, and sandbox profiles.
Lifecycle evidence: setup, start, readiness, reset, teardown, timeout, and skip behavior. A service definition without readiness and reset is not yet an agent-ready environment.
Real-only boundaries: credentials, paid or rate-limited APIs, signing, hardware, OS/kernel capabilities, proprietary data, and production-only topology.

Prefer executable configuration over prose when they conflict, and record the conflict instead of silently choosing one. Reuse the repository's narrowest working seam before inventing a parallel test architecture.

The discovery output is a boundary inventory:

Boundary	Behavior needed by claim	Existing seam	Candidate mode	Evidence gap
subject	changed behavior	production module	real	none
database	transactions + constraints	container fixture	ephemeral real	production topology
provider API	request/response contract	fake HTTP server	virtualized	live provider behavior
browser	DOM + origin behavior	browser fixture	real local browser	device/vendor integration

Choose Fidelity by Behavior

Use the lowest rung that preserves the behavior under judgment, not the lowest rung that makes the test green.

Verification claim	Minimum credible environment	What it does not prove
Pure transformation, validation, state machine	Real module + pinned inputs; no double unless needed to supply input	I/O integration
Calls, emitted events, retry selection	Fake/spy at the narrow collaborator boundary; keep orchestration real	Collaborator semantics
HTTP/RPC/message encoding and error handling	Local protocol server or virtual service with real serialization and request matching	Provider implementation
Consumer/provider compatibility	Consumer contract test plus provider verification; provider state explicit	End-to-end workflow
Database, broker, cache, filesystem semantics	Ephemeral real implementation or a compatibility-qualified emulator	Production scale/topology
Browser rendering, storage, origin, accessibility	Isolated real browser with local fixtures; virtualize only external network edges	Third-party/device integration
Core-path completion under degraded runtime capability	A pinned floor profile of the target runtime: constrained network, older engine version, missing platform API, failing native bridge	Behavior at full capability; pixel or layout fidelity
Compiler, debugger, OS, driver, native runtime	Real toolchain and target backend for the affected platform; use synthetic inputs only below that boundary	Other platform variants
Multi-service workflow	Real changed services plus contract-verified peers and real semantic state stores	Full production topology
Performance, races, resilience, security, vendor acceptance	Authorized sandbox/staging or controlled real boundary	Production behavior unless explicitly sampled

Use these decision rules:

If the real collaborator is cheap, deterministic, and safe in-process, keep it real.
If the claim depends on a collaborator's protocol only, virtualize the service but keep real serialization, parsing, and transport behavior.
If the claim depends on engine semantics, run the real engine ephemerally.
If an emulator has known compatibility limits, name them in the contract and add a calibration rung against the real boundary.
If no stable contract or authorized observation exists, do not invent the collaborator's behavior. Mark the required evidence unobserved or the run blocked.

“Real” is boundary-relative. A containerized PostgreSQL process is real for SQL and transaction semantics, but not for a managed service's topology, extensions, latency, failover, or IAM. A real local browser is real for DOM and origin behavior, but not for a payment provider's iframe or device wallet.

Capability degradation is a separate dimension from dependency fidelity: it asks how poor the target runtime may be, not how real the dependencies are. A floor profile is judged on whether the core path still completes — the flow reaches its terminal state and the persisted outcome is correct — and explicitly not on pixel or layout equality with the full-capability environment. Pin the floor (which engine version, which bandwidth and latency, which API absent, which bridge failing) in the environment contract, or the case silently re-tests the full-capability path.

For browser and UI claims, how the browser is driven is a separate choice from how real it is: an attached real-profile browser or an OS-driven user session has higher session realism but lower isolation than the L3 isolated real browser, and that inversion is recorded in the case constraints, not read as a higher rung. Use UI and System Drivers to choose the injection point and its observation adapter.

Fidelity Ladder

Build a ladder rather than one oversized environment. A change climbs only as high as its claim requires.

Rung	Environment	Typical evidence
L0 — preflight	Pinned toolchain, dependency/lock validation, configuration schema	versions, install/build readiness, manifest checks
L1 — hermetic component	Real subject, in-process inputs/doubles, no network	unit/property results, exact state/assertions
L2 — boundary simulation	Local protocol server, fake transport, contract fixture, recorded response	requests observed, protocol/error cases, contract artifact
L3 — semantic dependency	Ephemeral real database/broker/browser/toolchain or qualified emulator	migration/query/render/runtime behavior
L4 — sandbox system	Multiple real processes with virtualized external edges	workflow outcome and correlated evidence
L5 — real-boundary calibration	Authorized provider sandbox, device/platform matrix, staging sample	differential/contract result and remaining gaps

Every rung needs:

a bounded setup/start command;
a readiness probe that checks behavior rather than process existence;
unique or isolated state per run;
a deterministic reset and cleanup path, including failure cleanup;
a verification command with a machine-readable verdict;
a timeout and resource limit; and
an explicit list of higher-rung claims it cannot prove.

Do not run every rung by default. Map the change to the lowest sufficient rung, then escalate after a fail, unobserved, fidelity-sensitive diff, or scheduled calibration. Report the highest completed rung and every skipped required rung.

Bootstrap One Tracer Environment

Select one Agent Verify Loop tracer case and write its claim.
Build the boundary inventory from repository evidence.
Classify each dependency as real, ephemeral-real, emulator, virtual-service, fake, or unavailable; explain why.
Pin the toolchain, inputs, clock/seed, service versions, and platform.
Implement setup → readiness → reset → exercise → judge → teardown for the minimum credible rung.
Run a known-good control. It must pass with the required evidence present.
Run a known-bad control that breaks the behavior named in the claim. The harness must fail for the expected reason.
Calibrate the substituted boundary against a contract, real dependency, or authorized sandbox observation.
Only then generalize the environment for more cases or platforms.

The known-bad control is essential for AI-authored tests: it proves the harness is sensitive to the target behavior rather than merely executable.

Environment Contract

Store a declarative contract next to the cases or harness configuration. Keep commands portable; the YAML below is a shape, not a prescribed filename.

id: <stable environment id>
claim: <behavior this environment can prove>
source:
  revision: <repository commit or content hash>
  platforms: [windows, macos, linux] # actual supported subset
toolchain:
  runtime: <name + pinned version>
  dependencies: <lockfile or image digest>
subject:
  component: <real code under test>
  entrypoint: <command/module/route>
boundaries:
  - name: <dependency>
    mode: real | ephemeral-real | emulator | virtual-service | fake | unavailable
    contract: <schema/spec/provider test/observed invariant>
    version: <pinned version or digest>
    reason: <why this mode is sufficient for the claim>
    fidelity_gaps: [<behavior not preserved>]
lifecycle:
  setup: <bounded command>
  ready: <behavioral probe>
  reset: <clean-state command>
  stop: <cleanup command>
isolation:
  filesystem: <unique portable temp root>
  network: <disabled/allowlisted; dynamic ports preferred>
  state: <per-run namespace/database/profile>
  determinism: <clock, seed, locale, timezone, ordering>
oracle:
  source: <existing contract/reviewed baseline/provider invariant>
  command: <machine-readable judge>
  known_good: <control case>
  known_bad: <negative control and expected failure>
calibration:
  against: <real dependency/provider/sandbox or unavailable>
  command: <contract/differential check>
  last_observed: <timestamp + revision>
  expires_after: <policy>
constraints: [<authority, credentials, platform, resource limits>]

Do not store secret values or unsanitized production payloads in this asset. Record only secret names, providers, and required authority.

Make the Oracle Independent

An environment is not verified because the test process exited zero. The oracle must come from evidence not invented from the changed implementation. Prefer, in order:

a pre-existing acceptance test, public schema, language/runtime contract, or externally owned specification;
a consumer contract replayed against the real provider implementation;
a reviewed golden, fixture, trace, or snapshot with provenance;
an independently observed invariant from an authorized environment; or
a human-reviewed expected result for the initial tracer case.

Then prove sensitivity in both directions:

known good → pass with all required evidence collected;
known bad → fail at the expected assertion; and
unavailable evidence → unobserved, never pass.

Also inspect the test count, selected cases, skips, retries, and flaky results. 0 tests, “all relevant tests skipped,” or a runner configured to pass with no tests is not positive evidence. A newly generated baseline must not be approved by the same automated step that changed the implementation.

Control Drift

Every substitute drifts unless it is checked:

Version fakes and recorded responses with source, capture time, schema or provider revision, sanitizer version, and expiry policy.
Prefer contract fixtures with minimal required fields over copied full payloads, but keep unknown-field and error cases where compatibility depends on them.
Replay consumer contracts against the provider, or run a differential probe that sends the same sanitized cases to the substitute and authorized real boundary.
Exercise failures that are hard to summon from the real service: timeout, reset, malformed response, partial result, retryable and terminal errors.
Reset stateful virtual services between cases; hidden scenario state makes order-dependent tests.
When calibration expires or cannot run, preserve the last observation and demote the affected claim to unobserved or partial.

Record/replay is a bootstrap technique, not a permanent truth source. Redact before committing, parameterize volatile identifiers, and retain the contract that explains which parts of the recording matter.

Safety and Portability

Default to no production credentials, no unsanitized production data, and no unrestricted network. Use obviously fake test values or authorized, minimized fixtures.
Isolate agent-generated code in an ephemeral workspace, container, VM, or equivalent boundary appropriate to its risk. Do not treat a container as a complete security boundary.
Prefer dynamically allocated ports and platform temp-directory APIs over fixed ports and /tmp literals.
Pin runtime and dependency versions. For container images, pin a meaningful version and use a digest where reproducibility or supply-chain integrity requires it; avoid latest.
Make cleanup idempotent and run it after success, failure, and timeout. Give each parallel worker a unique state namespace.
Keep Windows, macOS, and Linux command/path differences in the lifecycle contract. A container-only path is one supported rung, not universal support.
Bound CPU, memory, disk, worker count, retries, and wall time. Resource exhaustion is blocked, not a product failure.

Cross-Stack Patterns Observed

The following patterns came from read-only Qoder CLI studies on 2026-08-04. No tests or services were run, so these are repository observations and design inputs, not runtime validation.

Repository shape	Observed environment strategy	Transferable lesson	Visible gap
Browser payment SDK (`braintree-web`)	Jest/jsdom, a central fake client/config factory, auto-mocked cross-frame transport, fake XHR, then build/export tests and real-browser stories	Replace the external transport, keep request construction, packaging transforms, DOM, and artifact contracts real	Unit fakes cannot prove cross-origin messaging, gateway, wallet, or sandbox behavior; bare Jest can make version assertions vacuous
Go debugger (`delve`)	Synthetic DWARF + fake memory for pure evaluation, in-memory network transport, recorded/core replay, then fixtures compiled by the real Go toolchain and real OS backends	Move the seam inward for fast tests, but keep compiler output and OS/kernel semantics real when they are the subject	Environment gates mostly skip; a green run without skip accounting overstates platform coverage
Spring microservices transaction sample	Real ephemeral PostgreSQL/RabbitMQ through Testcontainers, discovery disabled, and an in-process event bus supplying coordinator state	Mix substitutes by semantics; database/broker behavior can stay real while irrelevant topology is removed	Container/Compose versions drift, order-service lacks equivalent coverage, and unbounded waits can turn failures into hangs
TypeScript ecommerce monorepo (`nimara-ecommerce`)	Module-boundary HTTP/provider fakes with native requests, committed fake env values, real Postgres only for ledger work, LocalStack only for secrets work, deployed URL for E2E	Start with handler-level seams and add semantic services only for changed boundaries	`passWithNoTests` and unavailable E2E/codegen dependencies can create false-green or incomplete evidence

Evidence anchors inspected by the read-only studies:

braintree-web: test/helpers/index.js, __mocks__/framebus.js, test/publishing/setup.js, and .storybook/tests/helpers/test-server.ts.
delve: pkg/proc/dwarf_expr_test.go, service/listenerpipe.go, pkg/proc/test/support.go, and Documentation/backend_test_health.md.
Spring sample: account-service/src/test/kotlin/pl/piomin/samples/account/AccountControllerTests.kt, account-service/src/main/kotlin/pl/piomin/samples/account/service/EventBus.kt, transaction-server/src/test/kotlin/pl/piomin/samples/transaction/TransactionBrokerConfiguration.kt, and docker-compose.yml.
nimara-ecommerce: apps/marketplace/docker-compose.yml, apps/marketplace/src/app/api/stripe/connect/webhook/route.test.ts, apps/automated-tests/playwright.config.ts, and apps/storefront/vitest.config.ts.

Across all four shapes, the durable pattern is:

real changed subject
+ smallest contract-shaped substitutes
+ real semantics for the boundaries named in the claim
+ one higher-fidelity calibration path
+ explicit gaps and skip accounting

Anti-Patterns

Mock everything: the test proves the agent's assumptions about its own fakes, not the changed system.
Container first, claim later: a large Compose stack is expensive but may still omit the behavior under judgment.
AI-written oracle as sole evidence: implementation and expected result can share the same mistake.
Pass without sensitivity: no known-bad control proves the harness could catch the regression.
Green through zero or skipped tests: process success is mistaken for case coverage.
Ambient environment: local credentials, shared state, wall clock, fixed port, locale, or PATH silently decides the result.
Fake semantic engines: an in-memory map is treated as proof of database, broker, filesystem, or concurrency behavior.
Stale recordings: captured traffic has no provenance, redaction, contract, expiry, or provider replay.
Emulator equals production: known compatibility gaps disappear from the verdict.
Unavailable boundary hidden: a lower rung passes and the required higher rung is omitted from the report.

Research Basis

SWE-bench constructs a pinned execution environment per task and uses fail-to-pass behavior as the primary evaluation signal; its Multilingual validation procedure explicitly checks that tests fail before the gold change and pass after it.
SWE-agent uses ephemeral isolated execution and emphasizes concise command-effect feedback to the agent.
Testcontainers documents throwaway real databases, brokers, browsers, and application dependencies starting from known state.
Pact provider verification pairs a consumer-side mock interaction with replay against the real provider, preventing the mock from becoming the only contract authority.
WireMock service virtualization demonstrates controlled state, record/replay, latency, and fault injection at an HTTP boundary.
Playwright isolation uses a clean browser context per test to prevent state leakage and cascading failures.
Docker build guidance recommends pinning base-image versions and digests when supply-chain and reproducibility guarantees require it.