Autotests for agents

June 18, 2026 · View on GitHub

Guide for Cursor / Codex / CI bots: verify MailAgent on prod without human involvement (no real SMTP, no DATABASE_URL).

Operator connects secrets once → OPERATOR.md.

Quick start

npm ci

MAILAGENT_API_URL=https://api.webmailagent.com \
MAILAGENT_API_KEY=ma_… \
  npm run test:prod:gate   # CI after deploy (smoke only)

# Before merge or release:
  npm run test:prod        # full contracts + Playwright

CI after deploy runs test:prod:gate (smoke only — saves KV quota). Full suite: manual or tag v* publish workflow.

Test layers

LayerCommandWhereAPI key
Prod gate (CI)npm run test:prod:gatedeploy / PRMAILAGENT_API_KEY
Prod gate (full)npm run test:prodpre-merge, tag v*MAILAGENT_API_KEY
Smoke agentnpm run smoke:agentMCP, OAuth, DCR, Streamable HTTPyes
Smoke QAnpm run smoke:qainbox lifecycle on prodyes
Contract (all)npm run test:contract:all13 scripts via simulateyes
Playwright simulatenpm run test:pw:simulateCI gate, no DATABASE_URLyes
Typechecknpm run checkPR, no prodno
Codex scaffoldnpm run verify:codexPRno
Unit (local)npm run test:allowlist, test:extract, …dev, not prodno

Contract tests do not send real mail: messages are injected via POST /v1/inboxes/:id/simulate.

Environment variables

VariableRequiredValue
MAILAGENT_API_KEYyesteam key or legacy API_KEY
MAILAGENT_API_URLnodefault https://api.webmailagent.com
API_KEYfallbackalias for MAILAGENT_API_KEY
SMOKE_EXPECT_ATTACHMENTSsmoke:agent"1" in CI

Local key can live in .env — loaded by scripts/load-env.mjs.

Prod gate

CI (test:prod:gate) — scripts/test-prod-gate.mjs: smoke:agent → smoke:qa.

Full (test:prod) — scripts/test-prod.mjs:

Order:

  1. smoke:agent — discovery, OAuth metadata, DCR, MCP session, tool call
  2. smoke:qa — create → simulate → wait → extract → delete
  3. test:contract:all — all contract-qa scripts
  4. test:pw:simulate — Playwright simulate gate

Any failing step exits non-zero.

Contract scripts (one at a time)

Run a narrow script after changes in a specific area:

Scriptnpm scriptCovers
contract-qa.mjstest:contract:qacreate → simulate OTP → wait → extract
contract-qa-agent.mjstest:contract:qa:agentGET /v1/agent, /v1/me, /mcp/auth
contract-qa-agent-access.mjstest:contract:qa:agent-accessshort-lived scoped key broker
contract-qa-callback.mjstest:contract:qa:callbackwebhook callback after simulate; set CONTRACT_CALLBACK_URL for strict non-2xx failure
contract-qa-attachments.mjstest:contract:qa:attachmentsattachments + raw MIME
contract-qa-threads.mjstest:contract:qa:threadsthreads / reply grouping
contract-qa-domains.mjstest:contract:qa:domainscustom domains (Resend quota → skip)
contract-qa-search.mjstest:contract:qa:searchmessage search
contract-qa-extract.mjstest:contract:qa:extractstructured extract
contract-qa-console.mjstest:contract:qa:consoleconsole summary API
contract-qa-audit.mjstest:contract:qa:auditaudit log (async poll)
contract-qa-console-inbox.mjstest:contract:qa:console-inboxconsole inbox UI API
contract-qa-team-keys.mjstest:contract:qa:team-keysteam keys CRUD
contract-qa-session.mjstest:contract:qa:sessionrun session GET/PATCH
contract-qa-oidc.mjstest:contract:qa:oidcOIDC authorize redirect (skip if disabled)

Example — agent hub only after edits to src/routes/agent.ts:

MAILAGENT_API_KEY=ma_… npm run test:contract:qa:agent

CI (automatic runs)

WorkflowTriggerTests
deploy-worker.ymlpush main (Worker paths)deploy → test:prod:gate
qa-smoke.ymlPR / qa/**check + verify:codex + test:prod:gate
publish-packages.ymltag v*test:prod → npm publish

Without MAILAGENT_API_KEY in GitHub Secrets, deploy fails — by design.

Agent workflow after a code change

  1. Discovery — GET /v1/agent (tools, docs, auth).
  2. Types — npm run check (if you changed src/).
  3. Narrow contract — script from the table above.
  4. Full gate — npm run test:prod before merge / after deploy.
  5. Diagnostics — npm run doctor:qa (plan, outbound, oidc hints).

When a test fails

  1. Read stderr of the last contract script (name in --- contract-qa-….mjs ---).
  2. Re-run one script locally with the same key.
  3. Inbox flow: mailagent_diagnose_inbox or POST …/simulate via curl.
  4. Audit: event is async — contract already polls; if flaky, increase delay in the script.
  5. Domains: Resend quota — script cleanup + skip; not an API regression.
npm run doctor:qa
curl -s -H "Authorization: Bearer $MAILAGENT_API_KEY" \
  https://api.webmailagent.com/v1/agent | jq .

Adding a new contract test

  1. Copy template scripts/contract-qa.mjs.
  2. Use scripts/lib/contract-api.mjs (contractSimulate, contractApi).
  3. Register in scripts/test-contract-all.mjs.
  4. Add npm script test:contract:qa:<name> in package.json.
  5. Update this table and AGENTS.md.

Do not use DATABASE_URL or simulate-inbound.mjs in CI — HTTP simulate only.

E2E (Playwright / Vitest)

Product E2E with @mailagent/qa: QA.md and examples/playwright/.
Contract tests cover the API contract; Playwright covers external app UI.