OpenTester Architecture

March 8, 2026 · View on GitHub

This document describes the system architecture, design principles, and key architectural decisions of OpenTester.

Overview

OpenTester is an MCP-First Testing Execution Infrastructure designed for AI coding tools such as Claude Code, Cursor, and OpenCode. It provides a unified DSL format and MCP interface that enables Agents to generate, execute, and manage test cases.

The core workflow is: Agent generates DSL → OpenTester validates and executes → Results returned to Agent

High-Level Architecture

OpenTester follows a three-layer architecture:

┌─────────────────────────────────────────┐
│  AI Agent (Claude Code / Cursor / ...)  │
│  ├─ Generate DSL test cases             │
│  ├─ Decide testing strategies           │
│  └─ Analyze failure reasons             │
├─────────────────────────────────────────┤
│  OpenTester (MCP Server)                │
│  ├─ Validate DSL syntax                 │
│  ├─ Execute tests (CLI/Web)             │
│  ├─ Store cases/projects                │
│  └─ Return structured results           │
├─────────────────────────────────────────┤
│  Web UI (Auxiliary Observation Panel)   │
│  ├─ View execution progress             │
│  ├─ Debug cases (create/edit)           │
│  └─ View history reports                │
└─────────────────────────────────────────┘

Data flows from the Agent through OpenTester's MCP interface, with optional Web UI for human observation and debugging.

Design Principles

  • Agent Intelligence: Test generation and failure analysis are handled by the Agent
  • OpenTester Execution: Focuses on DSL validation and test execution
  • MCP-First: All core features exposed through MCP
  • Web UI Auxiliary: Visual monitoring and debugging, not required

System Boundaries

Current Capabilities

  • CLI Execution: Subprocess command execution (implemented)
  • Web Execution: Playwright-based browser automation (implemented)
  • MCP Interface: Streamable HTTP transport on port 8001
  • REST API: Auxiliary interface for Web UI on port 8000
  • Control Flow: if, loop, for_each conditionals (implemented)
  • Variable Resolution: ${name}, ${vars.name}, ${env.VAR} (no default-value syntax)
  • Failure Policies: fail, ignore, retry, fallback

Future Architecture (Phase 2/3)

  • GUI Testing: PyAutoGUI + AI vision (Phase 2)
  • TUI Testing: pexpect terminal interaction (Phase 3)
  • Extended Variables: ${{now}}, ${{random}}, ${{file:path}} (implemented)

Note: The execution engine currently supports CLI and Web. GUI and TUI execution targets are planned for future phases.

Backend Architecture

Entry Points

  • opentester start - Start both FastAPI and MCP services
  • opentester api - Start FastAPI server only (port 8000)
  • opentester mcp - Start MCP server only (port 8001)

Module Structure

backend/opentester/
├── main.py                  # FastAPI entry, CORS, router registration
├── models/                  # Pydantic models
│   ├── project.py          # TestProject, Target, TestGroup
│   ├── case.py             # TestCase model
│   ├── dsl.py              # DSLScript, TestStep, assertions
│   └── template.py         # DSLTemplate, TemplateVariable
├── core/                   # Core execution engine
│   ├── executors/
│   │   ├── base.py         # BaseExecutor, ExecutionContext
│   │   ├── cli.py          # CLIExecutor (subprocess)
│   │   └── web/executor.py # WebExecutor (Playwright + AI locator flow)
│   ├── execution_engine.py # ExecutionManager, lifecycle management
│   ├── storage.py          # ProjectStorage, ExecutionStorage, TemplateStorage
│   ├── global_state.py     # SQLite-based global execution state for multi-process sharing
│   └── config_manager.py   # Unified configuration manager with XDG compliance
│   ├── executors/
│   │   ├── base.py         # BaseExecutor, ExecutionContext
│   │   ├── cli.py          # CLIExecutor (subprocess)
│   │   └── web/executor.py # WebExecutor (Playwright + AI locator flow)
│   ├── execution_engine.py # ExecutionManager, lifecycle management
│   └── storage.py          # ProjectStorage, ExecutionStorage, TemplateStorage
├── api/                    # REST API (for Web UI)
│   ├── projects.py         # Project CRUD
│   ├── cases.py            # Test case management
│   ├── execution.py        # Execution endpoints, WebSocket
│   └── templates.py        # Template CRUD, instantiate
└── mcp/                    # MCP Server (PRIMARY INTERFACE)
    └── server.py           # FastMCP tools and handlers

MCP Server

The MCP server uses FastMCP from the official MCP SDK with Streamable HTTP transport (not SSE). Key design aspects:

  • Port: 8001 (configurable via FASTMCP_PORT)
  • Transport: Streamable HTTP
  • Direct Storage Access: Tools for projects/templates directly access storage
  • Execution Bridging: run_case, run_project, stop_execution, get_execution_status, and get_execution_log call FastAPI via HTTP to ensure executions are tracked in the main process
  • Direct AI DOM Tools: request_dom_analysis, submit_ai_selector, and list_paused_executions access execution manager/shared execution state directly

This creates an intentional coupling: FastAPI-bridged execution tools require the FastAPI server to be running.

Core Components

Global State Store (core/global_state.py)

SQLite-based global execution state for multi-process sharing between FastAPI and MCP servers:

  • WAL Mode: Write-Ahead Logging for concurrent read/write access
  • State Migration: Automatic migration from legacy JSON files
  • Diagnostic Limits: Configurable max diagnostic events per step/execution
  • Use Case: Enables MCP tools to query execution state without direct memory sharing

Configuration Manager (core/config_manager.py)

Unified configuration management with XDG Base Directory compliance:

  • XDG Compliance: Follows XDG_CONFIG_HOME and XDG_DATA_HOME standards
  • JSON-backed: Settings stored in ~/.config/opentester/settings.json
  • Singleton Pattern: Thread-safe singleton for consistent config access
  • Environment Override: Environment variables can override config values

Frontend Architecture

The Web UI is built with React + TypeScript + Vite and serves as an auxiliary observation panel.

Technology Stack

  • Framework: React with TypeScript
  • Build Tool: Vite
  • State Management: React Query for server state, Zustand for local state
  • Routing: React Router

Route Structure

RoutePurpose
/Dashboard
/projectsProject List
/projects/:idProject Detail
/projects/:projectId/cases/:caseId/execute/:executionId?Execution Monitor
/executions/:executionIdExecution Monitor (direct)
/projects/:projectId/batch/:batchIdBatch Execution Monitor
/executionsGlobal Executions List
/templatesTemplate List
/settingsSettings

Integration Points

  • REST API Client: src/lib/api.ts - calls /api/* endpoints
  • WebSocket: src/hooks/useExecutionWebSocket.ts - real-time execution updates
  • Dev Proxy: Vite config proxies /api → 8000, /mcp → 8001

Important: The Web UI is optional. All core features work through MCP without the Web UI.

Interface Architecture

MCP Interface (Primary)

  • Transport: Streamable HTTP
  • Endpoint: http://localhost:8001/mcp
  • Protocol: MCP 2024-11-05
  • Tool Categories:
    • Project Management (list_projects, get_project, create_project, delete_project)
    • Test Case Management (validate_dsl, save_case, delete_case)
    • Template Management (list_templates, create_template, instantiate_template, etc.)
    • Test Execution (run_case, run_project, stop_execution, get_execution_status)
    • AI DOM Analysis (request_dom_analysis, submit_ai_selector, list_paused_executions)

See MCP.md for complete tool documentation.

REST API (Auxiliary)

  • Base URL: http://localhost:8000/api
  • Purpose: Web UI support
  • CORS: Allows http://localhost:5173
  • Endpoints: Projects, Cases, Execution (with WebSocket), Templates, Settings

See API.md for complete endpoint documentation.

CLI Interface

  • Entry: opentester command
  • Commands: start, stop, status, doctor
  • Modes: Foreground with prefixed logs, API-only, MCP-only

See CLI.md for complete usage guide.

Execution and Data Flow

Execution Lifecycle

  1. Initiation: Agent calls MCP run_case or REST POST /api/execution/run
  2. State Creation: ExecutionManager.create_execution() creates state and persists to disk
  3. Step Execution: ExecutionManager._run_steps() iterates through DSL steps
  4. Browser Diagnostics: WebExecutor normalizes browser console/pageerror/requestfailed signals into persisted diagnostic_events linked to the active execution_id and step_index
  5. Real-time Updates: After each step, update_step() persists and notifies WebSocket subscribers; diagnostic events are also streamed to Web UI subscribers over the execution WebSocket
  6. Completion: set_status() marks completed/failed/stopped and persists final state

Key Components

  • ExecutionManager: Central coordinator managing execution state, persistence, and notifications
  • CLIExecutor: Command-line test execution
  • WebExecutor: Browser test execution with Playwright and AI-assisted locator support
  • ExecutionContext: Carries execution_id, project_id, case_id, target_type, variables

Failure Policies

Steps can specify on_failure behavior:

  • fail - Stop execution (default)
  • ignore - Mark step as skipped, continue
  • retry - Retry with configurable count, interval, and backoff (fixed/exponential)
  • fallback - Execute fallback steps

WebSocket Notifications

  • type: step_update - After each step completion
  • type: status_update - On status changes
  • type: initial - Full execution state on WebSocket connection
  • type: paused_for_ai - Execution paused waiting for AI selector
  • type: ai_selector_submitted - AI selector received, execution resumed

Persistence Strategy

  • Execution state is persisted after every step and status change
  • Browser diagnostic_events are persisted separately from step results so API clients can retrieve the raw event stream and aggregated summaries without rewriting historical execution records
  • SQLite shared state is used for cross-process execution coordination (including AI pause/resume)
  • JSON execution records under <XDG_DATA_HOME>/opentester/executions/{execution_id}.json remain for compatibility/history
  • ExecutionManager restores active/history state from persisted storage on startup

Data Architecture

Storage Location

OpenTester follows the XDG Base Directory Specification:

  • Config: ~/.config/opentester/ (or $XDG_CONFIG_HOME/opentester/)
  • Data: ~/.local/share/opentester/ (or $XDG_DATA_HOME/opentester/)

Note: Project/template/execution JSON data use XDG-compliant paths, while shared execution state for cross-process coordination uses ~/.opentester/state.db.

Directory Structure

~/.local/share/opentester/
├── projects/       # Project JSON files
├── executions/     # Execution record JSON files
└── templates/      # Template JSON files

Data Models

Project:

{
  "id": "uuid",
  "name": "Test Project",
  "target": {"type": "cli"},
  "cases": {
    "case-uuid": {
      "name": "Case Name",
      "dsl_content": "..."
    }
  }
}

Execution Record:

{
  "execution_id": "uuid",
  "case_id": "case-uuid",
  "project_id": "project-uuid",
  "status": "completed",
  "steps": [...],
  "created_at": "2026-02-26T10:00:00Z",
  "updated_at": "2026-02-26T10:01:00Z"
}

Template:

{
  "id": "uuid",
  "name": "Template Name",
  "description": "...",
  "target_type": "cli",
  "dsl_template": "version: \"1.0\"\n...",
  "variables": [...],
  "metadata": {...},
  "usage_count": 5
}

Variable Resolution

  • Syntax: ${vars.name} or ${env.VAR}
  • Direct var: ${name} is also supported
  • Resolution: BaseExecutor.resolve_variables()

Extended variables (${{now}}, ${{random}}, ${{file:path}}) are planned but not yet implemented.

Architecture Decision Records

ADR-1: MCP-First Architecture

Decision: Make MCP the primary interface, with REST API as auxiliary for Web UI.

Context: OpenTester integrates with AI Agents (Claude Code, Cursor, etc.) that need programmatic access. A standard protocol enables broad compatibility and decouples the interface from implementation.

Consequences:

  • ✅ Agents can directly invoke testing tools
  • ✅ Standard MCP protocol enables broad compatibility
  • ✅ Clear separation: Agents use MCP, humans use Web UI
  • ⚠️ Must maintain two interfaces (MCP + REST)
  • ⚠️ FastAPI-bridged execution tools depend on FastAPI availability

ADR-2: Removal of Internal AI Service

Decision: Remove the built-in AI service layer (ai_service.py, ai_providers/).

Context: The original design included an internal AI service for PRD parsing and test generation. This created duplication with Agent capabilities, added complexity for AI provider management, and caused context loss when the Agent explained intent to OpenTester.

Consequences:

  • ✅ Simpler codebase, fewer dependencies
  • ✅ Agent chooses model, prompt, strategy
  • ✅ Agent reasoning visible to user
  • ✅ Update Agent without code changes
  • ⚠️ Agent must understand DSL syntax
  • ⚠️ OpenTester validates but doesn't generate DSL

Note: Some code strings may still reference "AI-driven" as legacy artifacts.

ADR-3: Agent-Generated DSL

Decision: Agent generates DSL directly rather than OpenTester generating from PRD.

Context: The Agent has full context of the user's codebase and intent. Generating DSL directly removes PRD parsing complexity from OpenTester and makes the process transparent and editable.

Consequences:

  • ✅ Agent reasoning visible and editable
  • ✅ No PRD format constraints
  • ✅ Direct control over test generation
  • ⚠️ Agent must be trained on DSL syntax
  • ⚠️ Validation errors must be communicated back to Agent

ADR-4: Web UI as Auxiliary

Decision: Web UI is for observation and debugging, not the primary interface.

Context: MCP-first architecture means core features must work without Web UI. This enables headless deployments and aligns with the Agent-driven workflow.

Consequences:

  • ✅ Can run headless in CI/CD environments
  • ✅ All features work without browser
  • ✅ Simpler deployment options
  • ⚠️ Web UI maintenance is secondary priority
  • ⚠️ Some complex workflows may be easier in Web UI

Known Gaps and Future Architecture

Current Limitations

  1. GUI/TUI Execution: GUI and TUI are experimental and disabled by default
  2. Daemon Mode: Available via opentester start --daemon

Future Architecture

Phase 2 - GUI Testing:

  • PyAutoGUI integration for GUI automation
  • AI vision for element location
  • Visual cache for coordinate optimization

Phase 3 - TUI Testing:

  • pexpect integration for terminal interaction
  • ANSI parsing for terminal buffer analysis

Document Relationships

DocumentPurposeRelationship
README.mdUser-facing overview, installation, quick startEntry point for users
DEVELOPMENT.mdDeveloper guide, project structureImplementation details
MCP.mdComplete MCP tool referenceInterface documentation
API.mdComplete REST API referenceAuxiliary interface
CLI.mdCLI usage guideOperational commands
DSL_SPEC.mdDSL syntax specificationTest case format
SKILL_PROMPT.mdAgent integration guideAgent-specific guidance
CHANGELOG.mdVersion historyRelease notes
PRD.mdHistorical requirements (archived)Historical context only

Historical Context

OpenTester originally followed a PRD-driven approach: PRD documents were parsed by an internal AI service to generate test cases. This architecture was refactored to MCP-First to avoid duplicating Agent capabilities and to simplify the system. The current architecture places intelligence in the Agent and focuses OpenTester on execution infrastructure.

See PRD.md for the original vision and evolution rationale.