AI-Assisted Testing Playbook

May 11, 2026 ยท View on GitHub

This playbook describes how to use AI help without letting generated tests become the source of truth.

Workflow

  1. Decompose the requirement into behaviors first.
  2. Build a scenario matrix before generating test files.
  3. Identify the risk level for each scenario.
  4. Write explicit expected outcomes before accepting generated drafts.
  5. Add oracles that can fail a real mismatch.
  6. Review high-risk scenarios manually.
  7. Convert useful traces into regression scenarios.

Scenario Matrix

Cover the normal path, then add boundaries and abnormal paths:

  • expected tool use
  • forbidden tool use
  • approval required and approval not required
  • denied approval or blocked execution
  • timeout and retry behavior
  • malformed tool arguments
  • provider or gateway errors
  • permission and security boundaries
  • idempotency and duplicate requests
  • final state after the run

Reject Shallow Tests

Reject scenarios that only prove:

  • the runner started
  • a response existed
  • any tool was called
  • no exception was thrown

Useful scenarios say which tool should be called, which tool must not be called, what approval behavior is expected, and what final trace or answer evidence should exist.

Human Review

Use human review for high-risk flows such as shell execution, file writes, code execution, payments, home automation writes, MQTT publishes, and external integrations. A scenario generated by AI is a draft until those expectations and oracles are reviewed.

Trace to Regression Loop

When a real runtime or gateway adapter produces a useful trace:

  1. Redact secrets.
  2. Keep only stable evidence.
  3. Move important steps into scriptedTrace or a future replay fixture.
  4. Add explicit oracles for the behavior that mattered.
  5. Run openclaw test gates.
  6. Run openclaw test run --fail-on any.

The goal is not more tests. The goal is tests that make unsafe or incorrect behavior visible.