Try It Live

ai-natural-language-tests on Hugging Face Spaces — Try the platform directly in your browser without installation.

Table of Contents

Overview

The platform translates natural language requirements into executable E2E tests for:

flowchart LR
  A[AI-Powered E2E Test Generation]
  A --> C[Cypress<br/>.cy.js<br/>Traditional and prompt-powered]
  A --> P[Playwright<br/>.spec.ts<br/>TypeScript async/await]
  A --> W[WebdriverIO<br/>.spec.js<br/>Mocha with Jest-like expect]
  A --> M["Appium ⚠️<br/>.spec.js<br/>Mobile/iOS/Android"]

  style A fill:#e3f2fd,color:#333333,stroke:#666666
  style C fill:#c8e6c9,color:#333333,stroke:#666666
  style P fill:#ffcdd2,color:#333333,stroke:#666666
  style W fill:#ffe0b2,color:#333333,stroke:#666666
  style M fill:#f8bbd0,color:#333333,stroke:#666666

It supports both local engineering workflows and automated pipeline execution. The generator uses contextual data from live HTML analysis and historical pattern matching to produce stable, maintainable test assets.

Quick Start (5 Minutes)

git clone https://github.com/aiqualitylab/ai-natural-language-tests.git
cd ai-natural-language-tests
python -m venv .venv
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
npm ci
npx playwright install chromium

Create .env and set at least one provider key:

OPENAI_API_KEY=your_key

Generate and run one Playwright test:

python qa_automation.py "Test login with valid credentials" --url https://the-internet.herokuapp.com/login --framework playwright --run

Business Value

Core Capabilities

Architecture

graph TB
    subgraph "User Input"
        A[Natural Language<br/>Requirements]
        B[URL/HTML Data<br/>--url flag]
    C[CLI Requirement Text<br/>one or more prompts]
    end

    subgraph "AI & Workflow Engine"
        D[LangGraph Workflow<br/>5-Step Process]
        E[Multi-Provider LLM<br/>OpenAI / Anthropic / Google]
        F[Vector Store<br/>Pattern Learning<br/>FAISS + SQLite]
    end

    subgraph "Framework Generation"
        G{Cypress Framework}
        H{Playwright Framework}
        W{WebdriverIO Framework}
        I[Cypress Tests<br/>.cy.js files<br/>Traditional & cy.prompt&#40;&#41;]
        J[Playwright Tests<br/>.spec.ts files<br/>TypeScript]
        X[WebdriverIO Tests<br/>.spec.js files<br/>Mocha + expect]
    end

    subgraph "Execution & Analysis"
        K[Cypress Runner<br/>npx cypress run]
        L[Playwright Runner<br/>npx playwright test]
        M[AI Failure Analyzer<br/>--analyze flag<br/>Multi-Provider LLM]
        P[WebdriverIO Runner<br/>npx wdio run]
    end

    A --> D
    B --> D
    C --> D
    D --> E
    E --> F
    F --> D
    D --> G
    D --> H
    D --> W
    G --> I
    H --> J
    W --> X
    I --> K
    J --> L
    X --> P
    K --> M
    L --> M
    P --> M

    style D fill:#e3f2fd,color:#333333,stroke:#666666
    style E fill:#f3e5f5,color:#333333,stroke:#666666
    style F fill:#fff3e0,color:#333333,stroke:#666666
    style G fill:#c8e6c9,color:#333333,stroke:#666666
    style H fill:#ffcdd2,color:#333333,stroke:#666666
    style W fill:#ffe0b2,color:#333333,stroke:#666666

Workflow

flowchart TD
    A[Start: User Input<br/>Requirements + Framework] --> C[Step 2: Fetch Test Data<br/>Analyze URL/HTML<br/>Extract Selectors<br/>Generate Fixtures]
    C --> D[Step 3: Search Similar Patterns<br/>Query Vector Store<br/>Find Matching Test Patterns<br/>From Past Generations]
    D --> E[Step 4: Generate Tests<br/>Use AI + Patterns<br/>Create Framework-Specific Code<br/>Cypress, Playwright, or WebdriverIO]
    E --> H[HITL Approval<br/>Optional --approve before save]
    H --> F[Step 5: Run Tests<br/>Execute via Framework Runner<br/>Optional --run flag]
    F --> R[Replay Snapshot<br/>--list-html-replays / --replay-html-analysis]
    R --> G[End: Tests Executed<br/>Ready for CI/CD]

    style A fill:#e1f5fe,color:#333333,stroke:#666666
    style C fill:#c8e6c9,color:#333333,stroke:#666666
    style D fill:#ffcdd2,color:#333333,stroke:#666666
    style E fill:#f3e5f5,color:#333333,stroke:#666666
    style F fill:#e8f5e8,color:#333333,stroke:#666666
    style G fill:#f3e5f5,color:#333333,stroke:#666666

Generation follows a deterministic five-step flow:

Technology Stack

Repository Structure

ai-natural-language-tests/
|-- .github/
|-- docs/
|-- cypress/
|   |-- e2e/
|   |   |-- generated/
|   |   `-- prompt-powered/
|   `-- fixtures/
|-- tests/
|   `-- generated/
|-- webdriverio/
|   `-- tests/
|       `-- generated/
|-- web/
|-- webdriverio/
|   `-- tests/
|       |-- generated/
|       |-- appium-tests/
|       `-- prompt-powered/
|-- prompt_specs/
|-- skills/
|-- services/
|-- knowledge/
|-- generated_exports/
|-- rag_failure_analysis/
|-- vector_db/
|-- qa_automation.py
|-- qa_config.py
|-- qa_runtime.py
|-- qa_workflow.py
|-- qa_refinement.py
|-- test_qa_refinement.py
|-- ragas_evaluator.py
|-- ragas_nlp_evaluator.py
|-- cypress.config.js
|-- playwright.config.ts
|-- wdio.conf.js
|-- package.json
|-- package-lock.json
|-- requirements.txt
|-- Dockerfile
|-- docker-compose.yml
|-- agent.yaml
|-- SOUL.md
|-- RULES.md
|-- PROMPT_UPDATE_GUIDE.md
|-- CONTRIBUTING.md
|-- CHANGELOG.md
`-- README.md

Prerequisites

Installation

git clone https://github.com/aiqualitylab/ai-natural-language-tests.git
cd ai-natural-language-tests
python -m venv .venv
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
npm ci
npx playwright install chromium

OPENAI_API_KEY=your_key

$env:OPENAI_API_KEY = "your_key"

Optional: GitAgent (Repo-Specific)

This repository includes a targeted gitagent setup for its QA automation workflow:

• agent.yaml (manifest)
• SOUL.md and RULES.md (behavior and constraints)
• knowledge/ (framework and repo references)

In short: agent.yaml defines the repo agent, SOUL.md and RULES.md define how it should behave, and knowledge/ gives it project-specific framework guidance.

Quick commands:

npm run gitagent:validate
npm run gitagent:info
npm run gitagent:export

git clone https://github.com/aiqualitylab/ai-natural-language-tests.git
cd ai-natural-language-tests
docker compose build

docker compose run --rm test-generator "Test login" --url https://the-internet.herokuapp.com/login

docker compose run --rm test-generator \
  "Test login" --url https://the-internet.herokuapp.com/login --framework playwright --run

GitHub Registry (GHCR)

Pre-built Docker images are published to GitHub Container Registry. No local clone or build required.

Pull and run

docker pull ghcr.io/aiqualitylab/ai-natural-language-tests:latest

docker run --rm \
  -e OPENAI_API_KEY=your_key \
  ghcr.io/aiqualitylab/ai-natural-language-tests:latest \
  "Test login" --url https://the-internet.herokuapp.com/login

Image tags

For publishing and release management, see CONTRIBUTING.md.

Configuration

OPENAI_API_KEY=your_key
ANTHROPIC_API_KEY=your_key
GOOGLE_API_KEY=your_key

# Ollama (default: http://localhost:11434/v1)
OLLAMA_BASE_URL=http://localhost:11434/v1
OLLAMA_MODEL=llama2

# OpenAI-compatible endpoint (vLLM, LM Studio)
LOCAL_OPENAI_BASE_URL=http://localhost:8000/v1
LOCAL_OPENAI_MODEL=gpt-3.5-turbo

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --llm ollama
python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --llm local-openai

OTEL_PROVIDER=grafana
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-eu-north-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64(instance_id:api_token)>

GRAFANA_LOKI_URL=https://logs-prod-eu-north-0.grafana.net
GRAFANA_INSTANCE_ID=<instance_id>
GRAFANA_API_TOKEN=<logs_write_token>

Usage

Quick Reference

Demo & Test URLs

Use these URLs to try the platform immediately:

Quick test command:

python qa_automation.py "Test login with valid credentials" \
  --url https://the-internet.herokuapp.com/login \
  --framework playwright \
  --run

Natural Language Prompt Examples

Generate Cypress Test

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login

Generate Playwright Test

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework playwright

Generate WebdriverIO Test

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework webdriverio

Generate Appium Test (Experimental)

# Start Appium server in separate terminal
appium --port 4723 &

# Generate test (works without device)
python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework appium

Appium Setup & Execution (Experimental)

npm install -g appium@latest
appium driver install uiautomator2   # Android
appium driver install xcuitest       # iOS

# Windows PowerShell
$env:ANDROID_HOME = "C:\Users\YourUser\AppData\Local\Android\Sdk"
$env:PATH += ";$env:ANDROID_HOME\platform-tools"

# macOS/Linux
export ANDROID_HOME=~/Library/Android/sdk
export PATH=$PATH:$ANDROID_HOME/platform-tools

# Terminal 1: Start Appium server
appium --port 4723

# Terminal 2: Generate and run test
python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework appium --run

APP_PLATFORM=ios python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework appium --run

Prompt-Powered Cypress Mode

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --use-prompt

Generate and Execute

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login --framework playwright --run

Conversational Test Refinement

Refine generated tests in the UI with natural language instructions — multi-turn refinement supported.

1. Generate tests using the "Generate" button
2. Describe your refinement in the "Refine (describe the change)" textbox
3. Click "Refine" to apply the refinement
4. Repeat for multi-turn refinement

Refinement Examples:

Safety Guarantees:

• File integrity: LLM cannot add, remove, or rename files
• No empty tests: Every file must have runnable code
• No panel corruption: If refinement fails, original code is preserved
• Multi-turn support: Each refinement builds on the previous output

How It Works:

1. Generated code panel displays with // FILE: <path> headers
2. User enters refinement instruction (e.g., "also test the error message")
3. System sends current code + instruction to LLM
4. LLM returns complete revised code (never diffs or fragments)
5. Panel updates and files written to disk (if originals had headers)
6. Repeat for additional refinements

Failure Analysis

python qa_automation.py --analyze "CypressError: Element not found"
python qa_automation.py --analyze -f error.log

Test Code Review (AI Judge)

Review generated test code quality before execution. The AI judge scores tests on four dimensions and suggests concrete improvements.

Available in Gradio UI:

1. Generate tests using the "Generate" button
2. Review generated code in the "Generated Test Code" panel
3. Click "Review (AI judge)" button
4. Read scores and issues in "Review Scores" panel

Review Dimensions

Scoring & Verdict

• Each dimension scored 0–5
• "needs_work" verdict if ANY dimension scores ≤ 2
• "approve" verdict if all dimensions score > 2
• Issues list: up to 5 concrete problems, most important first
• Refinement suggestion: one-sentence fix for the biggest issue

Example Review Output

{
  "scores": {
    "assertions": 4,
    "selectors": 3,
    "structure": 5,
    "determinism": 2
  },
  "verdict": "needs_work",
  "issues": [
    "Add explicit waits before assertions to ensure determinism",
    "Use data-testid attributes instead of nth-child selectors",
    "Add error case assertion for failed login"
  ],
  "refinement_instruction": "Replace hardcoded timeouts with proper element waits and data-testid selectors."
}

Works with all frameworks

Code review applies uniformly to Cypress, Playwright, WebdriverIO, and Appium tests. The AI understands framework syntax from the framework context and provides actionable feedback.

Pattern Inventory

python qa_automation.py --list-patterns

AI Quality Evaluation (Ragas)

The project ships two evaluation scripts that measure whether the AI-generated tests are actually grounded in the page under test. They run automatically in CI but can also be run locally at any time.

Why these scripts exist

Generating test code from natural language is only useful if the generated tests match what the page actually contains. Without evaluation, you have no signal on whether GPT-4o-mini produced a test that talks about the right form fields, the right URL, or the right expected outcomes. These two scripts give that signal using Ragas, a framework designed to evaluate LLM outputs.

ragas_nlp_evaluator.py — No API key needed

This script compares the AI's generated text against reference answers using three classical NLP metrics that run fully offline.

The overall score for each sample is the average of all.

Run locally:

pip install ragas==0.4.3 sacrebleu rapidfuzz rouge-score
python ragas_nlp_evaluator.py

By default it reads test_dataset.json. Each entry in that file needs:

{
  "name": "login valid",
  "response": "The generated text from the AI",
  "reference": "The expected correct answer"
}

Pass a custom dataset or threshold:

python ragas_nlp_evaluator.py --dataset my_dataset.json --threshold 0.60

The script exits with code 1 if any sample scores below the threshold, which lets CI block on low-quality output. It also prints min, max, and overall average at the end.

ragas_evaluator.py — Requires OPENAI_API_KEY

This script evaluates the generated test file against the live page HTML using four LLM-based Ragas metrics.

It fetches the page, asks GPT-4o-mini to answer the requirement using page HTML, and scores how well that answer holds up.

Run locally:

python ragas_evaluator.py "Test user login with valid credentials" \
  --test cypress/e2e/generated/login.cy.js \
  --url https://the-internet.herokuapp.com/login

How they fit into CI

CI runs ragas_nlp_evaluator.py first (no API key, fast) as a quality gate. If it passes, the three framework jobs run in parallel — each generates tests, then calls ragas_evaluator.py against every generated file before running the tests themselves.

nlp-baseline job
  1. Install nlp dependencies
  2. Run ragas dataset baseline on test_dataset.json

if baseline passes
  test matrix jobs run in parallel
  1. Install dependencies
  2. Generate tests
  3. Run ragas generated test evaluation per file
  4. Run tests
  5. AI failure analysis if tests fail

CI/CD Integration

flowchart TD
  A[Code changes pushed to repo] --> B[Pipeline starts]
  B --> C[Install dependencies]
  C --> D[Run ragas dataset baseline]
  D --> E[Generate tests]
  E --> F[Run ragas generated test evaluation]
  F --> G[Run tests]
  G --> H[Tests pass]
  G --> I[Tests fail]
  H --> J[Deploy application]
  I --> K[Analyze failures]
  K --> L[Regenerate tests]
  L --> G
  K --> M[Notify developers]

Recommended pipeline stages:

Security and Compliance Guidance

Troubleshooting

{resource.service.name="ai-natural-language-tests"}

docker build --target debug -t ai-tests:debug .
docker run -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix ai-tests:debug

• Retry with generated single-spec command from logs.

Documentation Map

Versioning and Release Policy

Support and Security Reporting

Compliance and Data Handling

For implementation details and contribution controls, see CONTRIBUTING.md.

Operational Expectations

These are practical runbook-style expectations for delivery teams. They are operational targets, not contractual SLAs.

Support Matrix

The matrix below reflects currently configured and documented project baselines.

Changelog

Release notes are maintained in CHANGELOG.md, following the Keep a Changelog format.

Production-focused AI-assisted E2E test generation for modern QA teams.

© 2026 AI Quality Lab / Sreekanth Harigovindan
tests.aiqualitylab.org