简体中文

Waterfall AI Test

Open-source visual test workbench for Playwright Test Agents

Explore the workflow · Install the signed Beta · Understand the security boundary

https://github.com/user-attachments/assets/5aff4173-5871-4e76-a9bb-1460ab5ceb1a

▶ Watch the English introduction video (MP4)

Read: Human-in-the-Loop Playwright Test Generation

A visual workbench for Playwright Test Agents

Playwright Test Agents primarily deliver results through conversations and file directories. Waterfall AI Test organizes those Agent capabilities into a stateful, operable, and traceable testing workflow:

Conversation-driven Playwright Test Agents	Waterfall AI Test
Progress is hidden in chat history	Seven workflow stages and task states stay visible
Deliverables are scattered across file directories	Visually manage modules, plans, scripts, suites, and reports
Changes depend on continuing the conversation	View, edit, or regenerate directly in the UI
Interruptions require context to be explained again	Preserve stages, versions, tasks, and existing artifacts
Failure details are scattered across conversations and files	Connect logs, reports, videos, screenshots, and traces in one place

Public Beta: The current signed prerelease supports trusted, single-tenant Linux/amd64 Docker deployments only. Fresh installation only; not a hostile-code sandbox.

Waterfall AI Test is an independent open-source project built with Playwright. It is not affiliated with, sponsored by, or endorsed by Microsoft or the Playwright project.

Core capabilities

• One-click automation with human control: Let the Agent advance the full workflow automatically, or inspect, edit, regenerate, and take over at any stage.
• Requirements into test assets: Start from product requirements and build test modules, plans, UI exploration, Playwright scripts, and test suites.
• Real-browser verification: Run generated tests in a real browser and admit only verified scripts into the test suite.
• A closed failure-repair loop: Retain failure context, retry, repair, and reverify so every fix is confirmed by another execution.
• Versioned test assets: View and edit modules, plans, and scripts while local Git history supports traceability and restoration.
• Complete execution evidence: Associate every run with a summary, Run ID, logs, and test reports, plus videos, screenshots, and traces captured according to the Playwright configuration.

The interface and workflows are still evolving. See the roadmap, changelog, and support matrix before adopting the Beta. Code boundaries and extension rules are maintained in ARCHITECTURE.md.

Docker quickstart (source checkout)

This source-checkout quickstart is for isolated evaluation on Linux/amd64. It requires Docker Engine with Compose v2, Git, Python 3, and enough disk space for the images, browser, database, workspaces, and test artifacts. Use one consistent account for every command.

For signed Release verification, production-style deployment, backup, repair, and upgrade constraints, follow the complete deployment guide.

1. Prepare configuration

From the repository root:

cp deploy/config.example.json config.json
cp .env.example .env
chmod 600 config.json .env
./deploy/platform-compose init-config

init-config fills blank quickstart secrets without printing them and copies the database and OpenCode passwords into config.json. Both files now contain secrets: do not commit, paste, upload, or share them.

2. Validate and start

./deploy/platform-compose preflight-install
./deploy/platform-compose validate-config
./deploy/platform-compose up --build --detach
./deploy/platform-compose ps

The first build may take several minutes. This quickstart supports a fresh installation only; preflight-install rejects an existing runtime or Compose project.

3. Configure and verify a model Provider

Use OpenCode's interactive login, then select and verify an approved model:

./deploy/platform-compose opencode-auth-login
./deploy/platform-compose opencode-models PROVIDER_ID
./deploy/platform-compose opencode-set-model PROVIDER_ID/MODEL_ID
./deploy/platform-compose opencode-model-status
./deploy/platform-compose opencode-provider-smoke PROVIDER_ID/MODEL_ID

opencode-auth-login starts OpenCode's interactive Provider authentication flow. Select the Provider you want to use, then follow its prompts to sign in with OAuth or enter an API key. This follows the Provider-specific authentication flow documented by OpenCode Providers.

Replace PROVIDER_ID and MODEL_ID with an ID printed by opencode-models. opencode-auth-list lists stored Provider names without printing secret values, and opencode-model-status shows the configured global default. If OpenCode created a strict-JSON opencode.jsonc during startup, opencode-set-model safely migrates it to opencode.json while preserving its other settings. A customized JSONC file that uses comments or trailing commas is left unchanged and must be edited manually.

OPENCODE_SERVER_PASSWORD secures the platform-to-OpenCode HTTP service; it is not a model API key. Provider credentials and the global model configuration are stored in the persistent OpenCode data and config locations respectively. Use only a dedicated, least-privilege credential and do not copy its value into config.json or .env.

When the services are healthy, open http://127.0.0.1:5000 and sign in as admin. The password is the local PLATFORM_ADMIN_PASSWORD value in .env.

When you enter any project without a target system, the platform opens Project settings and prompts you to enter an authorized, isolated test-system URL. Save that setting and generate a Seed before running browser automation. Configure an approved model provider and complete one credential-free inference smoke test before relying on Agent features.

Useful commands:

./deploy/platform-compose verify
./deploy/platform-compose logs --follow platform
./deploy/platform-compose down

Trusted execution default: The bundled demo enables setup-script and Playwright execution. These capabilities run code and do not provide a hostile-code sandbox. Use only trusted users, repositories, and isolated test targets. For a public, shared, or otherwise untrusted deployment, set the following values in .env and recreate the platform service:

PLATFORM_ALLOW_HOST_SCRIPT_EXECUTION=false
PLATFORM_ALLOW_TEST_EXECUTION=false

./deploy/platform-compose up --detach --force-recreate platform

Configuration and secrets

config.json stores the OpenCode and platform database passwords. Target-system settings are maintained in Project settings and stored in the platform database, so config synchronization cannot overwrite UI changes. Keep both stores outside source control, restrict them to the service account, and include them only in encrypted, access-controlled backups.

Variable	Purpose
PLATFORM_SESSION_SECRET	Overrides the auth.session_secret file value
PLATFORM_ADMIN_PASSWORD	Overrides the auth.initial_admin_password file value
PLATFORM_DB_PASSWORD	Compose MySQL account password; copy the same value to platform_database.password
MYSQL_ROOT_PASSWORD	Compose MySQL bootstrap password
OPENCODE_SERVER_PASSWORD	Compose OpenCode service password; copy the same value to opencode_password
PLATFORM_COOKIE_SECURE	Set to true behind HTTPS
PLATFORM_ALLOW_TEST_EXECUTION	Allows generated Playwright code; enabled by the bundled demo defaults
PLATFORM_ALLOW_HOST_SCRIPT_EXECUTION	Allows trusted setup shell code; enabled by the bundled demo defaults

The Generate Seed menu offers a visit-only mode and a login mode. A visit Seed uses a fixed script that only opens base_url; it creates no model job and does not write the login URL, username, or password into the Seed. A login Seed keeps the model-generated login flow and provides the configured login details to the model. Both modes replace the same tests/seed/seed.spec.ts file.

The platform may still include target-system usernames and passwords in planning or script-generation prompts and login Seeds. Use only disposable, least-privilege credentials for an isolated non-production target. Treat the model provider, workspace Git history, logs, and execution artifacts as being inside that credential's exposure boundary.

The Compose quickstart fixes the MySQL database and application account to the values in deploy/config.example.json. To customize either identifier, update both the mysql service in deploy/compose.yaml and the platform_database.database / platform_database.user fields in the copied root config.json; changing only one side prevents the platform from connecting.

See configuration.md for the full schema and precedence rules.

Security boundary

The supported deployment assumes:

• one organization and one trust domain;
• trusted operators, platform users, repositories, generated code, and setup scripts;
• one application instance on Linux/amd64 Docker;
• an authorized, isolated, recoverable non-production target system;
• restricted network access to MySQL, OpenCode/model services, and the target;
• TLS and an external access-control layer in front of the platform.

Playwright tests and setup scripts execute code. They share the application container's operating-system boundary and are not isolated from the platform as hostile code. Execution switches and container restrictions can reduce accidental exposure; they do not turn the container into a sandbox. Do not give untrusted users execution permissions, mount a Docker socket, or connect the platform to production credentials or production data.

Read the complete security model and deployment guide.

Database baseline: file mode only

The legacy command baseline mode and the old backup.bat / restore.bat helpers are removed. database_baseline accepts only mode: "file" and copies a file-backed test database:

• if baseline_path does not exist, the current database_path is copied to create it;
• otherwise the baseline is copied back to the runtime database before the relevant test flow;
• a lock directory serializes the operation.

Example:

{
  "database_baseline": {
    "enabled": true,
    "mode": "file",
    "database_path": "/data/playwright-projects/default/data/test.db",
    "baseline_path": "/data/playwright-projects/default/.baseline/test.db",
    "lock_path": "/data/playwright-projects/default/.baseline/restore.lock",
    "timeout_seconds": 300
  }
}

Do not point this feature at production data. Server databases such as MySQL are not command-baseline targets; use a separately reviewed setup workflow and least-privilege test credentials if reset automation is required.

Sensitive artifacts

Playwright reports, screenshots, video, trace, browser downloads, workspace Git history, logs, and diagnostic bundles can contain cookies, form values, personal information, page content, or internal URLs. There is no reliable automatic sanitizer for every text field or binary artifact.

• restrict artifact access to trusted users;
• define retention and deletion limits;
• encrypt backups and keep mysql_data, platform_projects (including file baselines), platform_workspaces and every workspace Git history, all four OpenCode XDG volumes, and config.json/.env/Release metadata as one recovery point;
• review every diagnostic bundle before sharing it;
• never attach raw reports, trace, video, configuration, or database dumps to a public Issue.

Brand and release compatibility

The project was renamed from playwright-test-platform to waterfall-ai-test-platform. GitHub redirects the old repository URL. The immutable v0.1.0-beta.3 release intentionally keeps its original playwright-test-platform-* asset names and ghcr.io/jiongfeng/playwright-test-platform image reference; releases created after the rename use the Waterfall AI Test names. Runtime compatibility identifiers such as the playwright_platform database, Python package paths, container paths, and existing session-cookie names remain unchanged.

Fresh-install-only upgrade boundary

The current Public Beta supports fresh installation only. It does not support an in-place upgrade from an internal package, a legacy installation, a source checkout, or an installation whose release metadata is missing or unknown. The old internal incremental packages are retired and are not release assets or a public compatibility contract.

• install a Release bundle only into a destination that does not exist (confirm and explicitly remove an empty directory first), with new database and application volumes;
• do not point the candidate or future Release at a legacy database, workspace, Compose project, or volume;
• the read-only deploy/preflight-install.py check and deploy/upgrade-matrix.json deny every unlisted source; the current matrix has no supported in-place path;
• do not bypass a denial by deleting only a version marker or by relabeling an old image;
• preserve the old environment as one encrypted recovery point containing mysql_data, platform_projects, platform_workspaces and every workspace Git history, all four OpenCode XDG volumes, and config.json/.env/Release metadata. The project does not currently provide a public legacy export/import tool.

The release-bundle installer runs this read-only check before writing its destination. To audit the decision manually from an unpacked release bundle:

python3 deploy/preflight-install.py \
  --target /srv/waterfall-ai-next \
  --release-metadata ./RELEASE-METADATA.json

Exit status 10 is a policy denial. The check also denies a fresh install when the Compose project already owns containers, volumes, or networks. Do not work around a denial; use a genuinely separate destination and resource set.

Rotate any secret that appeared in a legacy configuration, script, log, archive, database backup, image, or Git history before manually re-entering approved settings in the clean environment. Deleting a secret from the latest file does not remove historical copies.

Local development and demo workspace

The credential-free examples/demo-workspace contains one in-memory Playwright test and no node_modules. The root config.example.json points to it for local development. Docker uses the separate deploy/config.example.json and a persistent workspace volume.

Backend tests:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python -m unittest discover -s tests -p 'test_*.py' -v

Demo test:

cd examples/demo-workspace
npm ci
npx playwright install chromium
npm test

Do not use real credentials or production targets in development tests.

Repository map

app.py                  compatibility composition root
test_plan_viewer/       domain, web, repository, and infrastructure modules
static/                 browser code and styles
templates/              Jinja templates
project-template/       generated workspace template
examples/demo-workspace credential-free local example
deploy/                 Docker, install preflight, upgrade policy, and health checks
docs/                   architecture, configuration, deployment, and security
tests/                  Python and JavaScript regression tests

Contributing, support, and security

• Read CONTRIBUTING.md and the Code of Conduct before opening a pull request.
• Use SUPPORT.md for reproducible, sanitized usage reports.
• Report vulnerabilities privately as described in SECURITY.md. Do not disclose credentials or unpatched vulnerability details in public Issues or pull requests.
• Project decisions follow GOVERNANCE.md.

Source code, including the checked-in project template and demo workspace, is licensed under the Apache License 2.0. When the platform copies the template to create a user workspace, it marks that generated workspace private and UNLICENSED so its owner can choose an appropriate license.