OpenTester Development Guide

March 8, 2026 · View on GitHub

This document provides guidance on project architecture, code conventions, and development workflow for developers.

Architecture Overview

┌─────────────────────────────────────────┐
│  AI Agent (Claude Code / Cursor / ...)  │
│  ├─ Generate DSL test cases             │
│  ├─ Decide testing strategies           │
│  └─ Analyze failure reasons             │
├─────────────────────────────────────────┤
│  OpenTester (MCP Server)                │
│  ├─ Validate DSL syntax                 │
│  ├─ Execute tests (CLI/Web)             │
│  ├─ Store cases/projects                │
│  └─ Return structured results           │
├─────────────────────────────────────────┤
│  Web UI (Auxiliary Observation Panel)   │
│  ├─ View execution progress             │
│  ├─ Debug cases (create/edit)           │
│  └─ View history reports                │
└─────────────────────────────────────────┘

Core Principles

  • Agent Intelligence: Test generation and failure analysis handled by Agent
  • OpenTester Execution: Focuses on DSL validation and test execution
  • MCP-First: All core features exposed through MCP
  • Web UI Auxiliary: Visual monitoring and debugging, not required

Project Structure

backend/opentester/
├── main.py                  # FastAPI entry, CORS, router registration
├── models/                  # Pydantic models
│   ├── project.py          # TestProject, Target, TestGroup
│   ├── case.py             # TestCase model
│   ├── dsl.py              # DSLScript, TestStep, assertions
│   └── template.py         # DSLTemplate, TemplateVariable
├── core/                   # Core execution engine
│   ├── executors/
│   │   ├── base.py         # BaseExecutor, ExecutionContext
│   │   ├── cli.py          # CLIExecutor (subprocess)
│   │   └── web/executor.py # WebExecutor (Playwright + AI DOM analysis)
│   ├── execution_engine.py # ExecutionManager, lifecycle management
│   └── storage.py          # ProjectStorage, ExecutionStorage, TemplateStorage
├── api/                    # REST API (for Web UI)
│   ├── projects.py         # Project CRUD
│   ├── cases.py            # Test case management
│   ├── execution.py        # Execution endpoints, WebSocket
│   └── templates.py        # Template CRUD, instantiate
└── mcp/                    # MCP Server (PRIMARY INTERFACE)
    └── server.py           # JSON-RPC handlers, tool definitions

Development Environment

Start Services

cd backend

# Start all services
uv run opentester start

# Start individually
uv run opentester api   # FastAPI (port 8000)
uv run opentester mcp   # MCP (port 8001, Streamable HTTP transport)

Port Assignment

MCP Implementation

Tool Definition

@mcp.tool()
def validate_dsl(dsl_yaml: str) -> str:
    """Validate DSL YAML syntax and schema.

    Args:
        dsl_yaml: DSL YAML content to validate
    """
    # Validation logic
    return json.dumps(result)

Adding New Tools

  1. Add tool function in opentester/mcp/server.py
  2. Use @mcp.tool() decorator
  3. Return readable text or structured output consistent with existing tools
  4. Add docstring describing parameters

Data Storage

OpenTester follows the XDG Base Directory Specification:

  • Linux: ~/.local/share/opentester/
  • macOS: ~/Library/Application Support/opentester/
  • Windows: %LOCALAPPDATA%\opentester\

Project Data

Stored in <XDG_DATA_HOME>/opentester/projects/:

{
  "id": "uuid",
  "name": "Test Project",
  "target": {"type": "cli"},
  "cases": {
    "case-uuid": {
      "name": "Case Name",
      "dsl_content": "..."
    }
  }
}

Execution Records

Stored in <XDG_DATA_HOME>/opentester/executions/{execution_id}.json:

{
  "execution_id": "uuid",
  "case_id": "case-uuid",
  "project_id": "project-uuid",
  "status": "completed",
  "steps": [...],
  "created_at": "2026-02-26T10:00:00Z",
  "updated_at": "2026-02-26T10:01:00Z"
}

Execution records are persistent and retained after server restart.

Template Data

Stored in <XDG_DATA_HOME>/opentester/templates/{template_id}.json:

{
  "id": "uuid",
  "name": "Template Name",
  "description": "Template description",
  "target_type": "cli",
  "dsl_template": "version: \"1.0\"\n...",
  "variables": [
    {"name": "base_url", "description": "API base URL", "required": true, "default_value": "http://localhost:8000"}
  ],
  "metadata": {
    "category": "api",
    "tags": ["health", "monitoring"],
    "author": "",
    "version": "1.0"
  },
  "usage_count": 5,
  "created_at": "2026-02-27T10:00:00Z",
  "updated_at": "2026-02-27T10:00:00Z"
}

Templates use ${variable_name} as variable placeholders, replaced with actual values during instantiation.

Variable Resolution

Variables use ${name}, ${vars.name} or ${env.VAR} syntax:

  • Resolved in BaseExecutor.resolve_variables()
  • Default-value syntax like ${vars.timeout:30} is not implemented

Web Testing and AI DOM Analysis

  • TargetType.WEB is supported and routed to WebExecutor.
  • WebExecutor supports web actions such as launch, close, navigate, click, type, select, wait, screenshot, assert.
  • If launch is not specified explicitly, browser initialization is performed automatically on the first non-close web action.
  • With ai_locator in a step, execution can enter paused_waiting_for_ai.
  • AI pause/resume REST endpoints:
    • GET /api/execution/{execution_id}/ai-pending-state
    • POST /api/execution/{execution_id}/submit-selector
    • GET /api/execution/paused/list
  • MCP AI tools:
    • request_dom_analysis
    • submit_ai_selector
    • list_paused_executions

Extended Variables

Extended variables are supported by executor interpolation:

  • ${{now}} - Current time (ISO 8601)
  • ${{random}} - Random hex string
  • ${{file:path}} - File content

Commit Conventions

  • feat: New feature
  • fix: Bug fix
  • docs: Documentation
  • refactor: Refactoring
  • test: Tests