Awesome AI Testing [](https://awesome.re) [](https://github.com/tugkanboz/awesome-ai-testing/actions/workflows/lint.yml)

July 14, 2026 · View on GitHub

A curated list of AI-powered testing tools, frameworks, and resources for QA engineers.

AI is reshaping software testing. This list collects tools, platforms, and resources that use AI or LLMs to generate tests, heal broken locators, triage failures, write assertions in natural language, and more. Both open source and commercial offerings are included, marked with badges so you can filter by what fits your stack.

Contents

Legend

  • 🆓 Open source
  • 💰 Commercial
  • 🆓💰 Open core (free tier or open source with paid features)

Star History

Star History Chart

Test Generation

Tools that generate test cases from code, requirements, or user behavior using AI.

  • Qodo-Cover 🆓 - AI-powered tool for automated test generation and code coverage enhancement.
  • EvoMaster 🆓 - First open source AI tool that automatically generates test cases via evolutionary algorithms for REST, GraphQL, and RPC APIs.
  • EvoSuite 🆓 - Generates JUnit tests using evolutionary and genetic search-based algorithms for Java.
  • Pynguin 🆓 - Automated unit test generation for Python via evolutionary algorithms, from Saarland University.
  • Symflower 💰 - Commercial tool that combines symbolic execution, static analysis, and LLMs to generate unit tests for Java, Go, and Kotlin. Free CLI available with limitations.
  • CodiumAI / Qodo 💰 - AI assistant generating meaningful tests from code context.
  • Diffblue Cover 💰 - Autonomous Java unit test writer using reinforcement learning.
  • GitHub Copilot 💰 - AI pair programmer that generates test code in Playwright, Cypress, Selenium across editors.
  • Cursor 💰 - AI-first code editor with strong test generation capabilities for major frameworks.
  • Claude Code 💰 - Anthropic's terminal-based agentic coding assistant, useful for test suite generation and refactoring.

MCP-Based Testing

Tools and servers that use the Model Context Protocol to give AI agents browser control and testing capabilities.

  • Playwright MCP 🆓 - Official Playwright MCP server giving AI agents full browser control through structured accessibility snapshots.
  • Playwright CLI 🆓 - Token-efficient CLI for coding agents like Claude Code and GitHub Copilot, with installable skills.
  • Chrome DevTools MCP 🆓 - Official MCP server from the Chrome DevTools team, with 26 tools for browser automation, debugging, and performance analysis.
  • ExecuteAutomation Playwright MCP 🆓 - Community Playwright MCP server with API testing support and 143 device emulation profiles.
  • BrowserTools MCP 🆓 - Browser monitoring and console log access for AI agents via Chrome extension.
  • Browser MCP 🆓 - Popular MCP server that automates the user's own local browser, preserving logged-in sessions and avoiding bot detection. Note: limited maintenance activity since mid-2025 but widely used (6.5k+ stars).
  • Podium MCP 🆓 - MCP server purpose-built for mobile app testing on Android/iOS simulators using Maestro, with deep Redux state inspection and CI pipeline integration.
  • Puppeteer MCP 🆓 - Reference MCP server for Puppeteer-based browser automation from the official MCP servers repo.
  • prufa-mcp 🆓💰 - Open-source (Apache-2.0) MCP server connected to Prufa's hosted audit backend; an AI agent runs a QA audit of a web app (analytics, broken flows, security headers, accessibility) and gets back machine-verified findings. First audit is free; further audits and features require a paid plan.

Self-Healing Test Frameworks

Tools that automatically repair broken test locators and adapt to UI changes.

  • Healenium 🆓 - Self-healing library for Selenium, Appium, and Playwright. Replaces broken selectors at runtime.
  • CodeceptJS 🆓 - End-to-end testing framework with built-in AI heal plugin that uses OpenAI, Anthropic, or local models to repair failing steps and propose locator fixes.
  • Testim 💰 - Pioneer of self-healing tests with AI-driven smart locators.
  • Functionize 💰 - AI-powered tests that adapt without selectors.
  • TestSigma 💰 - AI-driven low-code platform with self-healing across web, mobile, and API.
  • Tricentis Tosca 💰 - Enterprise platform with Vision AI for resilient automation.
  • Perfecto 💰 - Cloud testing platform with self-healing locators.

AI-Powered E2E Platforms

End-to-end testing platforms with AI at the core.

  • TestZeus Hercules 🆓 - World's first open-source testing agent for UI, API, security, accessibility, and visual validations, no code required.
  • agent-qa 🆓 - Self-improving QA agent for natural-language web and mobile tests with run memory, UI-change adaptation, and regression detection.
  • Mabl 💰 - Low-code platform with auto-healing and ML-based test maintenance.
  • Meticulous 💰 - Records real user sessions and generates regression tests automatically.
  • Momentic 💰 - AI-native end-to-end testing platform that writes, runs, and maintains web and mobile tests automatically using natural language.
  • Autify 💰 - No-code end-to-end testing platform with AI-driven maintenance.
  • Reflect 💰 - No-code regression testing with AI-assisted authoring.
  • QA Wolf 💰 - AI-powered QA-as-a-service generating Playwright tests at scale.
  • Bug0 💰 - Agentic testing platform built on the Planner, Generator, Healer pattern with MCP integration.
  • Checksum 💰 - Generates Playwright and Cypress tests from real user sessions.
  • Rainforest QA 💰 - No-code testing platform with AI-driven test generation.
  • BrowserStack Low Code Automation 💰 - AI-powered low-code platform with self-healing agents, NL test authoring, and real device cloud execution.
  • LambdaTest KaneAI 💰 - GenAI-native test agent for authoring, executing, and maintaining tests using natural language.
  • Virtuoso QA 💰 - NLP-based codeless E2E platform with self-healing and visual testing.
  • Katalon Studio 🆓💰 - Test automation platform with AI features including TrueTest and Visual Testing.

Mobile AI Testing

AI-powered tools specifically for mobile app testing.

  • Maestro 🆓💰 - YAML-based mobile UI automation that reads accessibility tree, no XPath needed.
  • Appium 🆓 - Industry standard mobile automation, with AI plugins for self-healing.
  • Sofy 💰 - No-code AI mobile testing platform.
  • Kobiton 💰 - Mobile device cloud with AI-driven scriptless automation.
  • HeadSpin 💰 - Mobile testing platform with AI-driven performance insights.
  • Waldo 💰 - Scriptless mobile testing platform with visual regression and AI-driven flow recording.
  • testRigor 💰 - Plain English test authoring across web, mobile, and API with AI-driven element identification.

Visual AI Testing

Visual regression and UI verification powered by AI.

  • Lost Pixel 🆓💰 - Open source visual regression testing.
  • Loki 🆓 - Visual regression testing for Storybook.
  • Argos 🆓💰 - Open source visual testing for engineering teams.
  • BackstopJS 🆓 - Visual regression testing for responsive web UIs.
  • Pixelmatch 🆓 - Pixel-level image comparison library.
  • Reg-Suit 🆓 - Visual regression testing workflow with publish/notify integrations for CI.
  • Applitools Eyes 💰 - Visual AI platform with cross-browser and cross-device verification.
  • Percy 💰 - Visual review and regression testing, part of BrowserStack.
  • Chromatic 💰 - Visual and interaction tests for Storybook.
  • Happo 💰 - Cross-browser screenshot testing with Playwright, Cypress, and Storybook integrations.

Natural Language Test Authoring

Write tests using plain English (or other natural languages).

  • Shortest 🆓 - QA via natural language AI tests, built on Playwright.
  • Magnitude 🆓 - AI-native, vision-first testing framework that lets you write E2E tests in plain language.
  • Auto Playwright 🆓 - Run Playwright tests with AI through plain text prompts.
  • Passmark 🆓 - Open-source AI regression testing framework on Playwright with intelligent caching, auto-healing, and multi-model verification.
  • Midscene.js 🆓 - AI-driven UI automation with natural language commands.
  • ZeroStep 💰 - Plain English test steps that compile to Playwright actions.
  • Tusk 💰 - AI agent that auto-generates and runs tests for pull requests.

LLM-as-Judge Evaluation

Use LLMs to evaluate test outputs, assertions, and quality.

  • Promptfoo 🆓💰 - Test framework with LLM-as-judge for prompts, models, and RAG pipelines.
  • DeepEval 🆓💰 - Pytest-like LLM evaluation framework with built-in judge metrics.
  • Ragas 🆓 - Evaluation framework for RAG pipelines using LLM judges.
  • Inspect AI 🆓 - LLM evaluation framework from the UK AI Safety Institute.
  • TruLens 🆓 - Evaluation framework for LLM apps with feedback functions and tracing.
  • Arize Phoenix 🆓 - Open-source LLM observability and evaluation.
  • Weights & Biases Weave 🆓💰 - Weights & Biases toolkit for tracing, debugging, and evaluating generative AI applications with built-in LLM-as-judge metrics and dataset management.
  • OpenAI Evals 🆓 - Framework for evaluating LLMs and an open-source registry of benchmarks from OpenAI. No longer actively maintained for new evals, but still widely used as a reference.
  • lm-evaluation-harness 🆓 - EleutherAI's framework for few-shot evaluation of language models, backing the Hugging Face Open LLM Leaderboard.
  • Langfuse 🆓💰 - Open source LLM observability, tracing, and evaluation platform.
  • Helicone 🆓💰 - Open source LLM observability and prompt evaluation platform.
  • Opik 🆓💰 - Open-source LLM evaluation and observability platform with automated tracing, LLM-as-judge metrics, and pytest integration for CI pipelines.
  • LangWatch 🆓💰 - Open-source LLM evaluation and AI agent testing platform combining end-to-end scenario simulation, observability, and prompt management in a single unified loop.
  • Evidently 🆓💰 - Open-source Python library for evaluating, testing, and monitoring ML and LLM systems with 100+ built-in metrics for data quality, drift detection, and LLM output quality.
  • LangSmith 💰 - LangChain's platform for testing and monitoring LLM apps.
  • Braintrust 💰 - LLM eval platform with experiments, datasets, and observability.
  • Patronus AI 💰 - Automated evaluation and security testing for LLMs.
  • Respan 🆓💰 - LLM observability, evaluation, and prompt management platform with a multi-provider gateway.

Test Analytics and Triage

AI for failure analysis, flaky test detection, and reporting.

  • agenttrace 🆓 - Local-first TUI and CLI for evaluating AI coding agent sessions with cost, token, latency, and failure regression gates.
  • ReportPortal 🆓💰 - Open source results management with ML-based failure clustering.
  • Allure TestOps 💰 - Test management with AI-driven analytics and flaky detection.
  • Sealights 💰 - Quality intelligence platform using ML for test gap analysis.
  • Trunk Flaky Tests 💰 - ML-based flaky test detection and quarantine.
  • Datadog CI Visibility 💰 - Test analytics with flaky test detection across CI pipelines.
  • Launchable 💰 - ML-driven predictive test selection and flaky test detection.
  • BuildPulse 💰 - Flaky test detection and analytics platform.
  • flaky-test-detector 🆓 - AI agent skill that detects, analyzes, and eliminates flaky tests across any test runner. Compatible with opencode, Claude Code, Cursor, Windsurf, and GitHub Copilot.

Code Coverage with AI

Tools that use AI to fill coverage gaps and prioritize testing efforts.

  • Mutahunter 🆓 - LLM-based mutation testing for stronger test suites.
  • Stryker Mutator 🆓 - Mutation testing framework that pairs well with AI test generators.
  • coverage-guard 🆓 - AI agent skill that enforces 100% test coverage for any JavaScript/TypeScript project. Works with Vitest, Jest, react-scripts, and more. Compatible with opencode, Claude Code, Cursor, Windsurf, and GitHub Copilot.

AI Test Data Generation

Tools that use AI to generate realistic test data, fixtures, and edge cases.

  • Faker.js 🆓 - Standard fake data library, often paired with AI for context-aware data.
  • Datafaker 🆓 - Java/Kotlin port of Faker for realistic fake data generation in JVM-based test suites.
  • Snowfakery 🆓 - Relational synthetic data generator from Salesforce, useful for complex schema-aware test fixtures.
  • Mockaroo 🆓💰 - Realistic test data generation with AI-suggested schemas.
  • Synthesized 💰 - AI-driven synthetic data platform for testing.
  • Tonic.ai 💰 - Generate realistic safe test data from production using AI.
  • Gretel 💰 - Synthetic data platform with AI-generated test datasets.

Mock and Service Virtualization

Tools for mocking external services, LLM APIs, and dependencies in AI testing pipelines.

  • Mountebank 🆓 - Mature open source service virtualization for HTTP, HTTPS, TCP, and SMTP protocols. Supports stubbing, mock verification, and record-playback.
  • WireMock 🆓💰 - Industry standard HTTP mocking with 7.1k stars and a native MCP server in the cloud version for AI coding assistants.
  • Mockoon 🆓 - GUI-based open source API mocking tool, easy to set up.
  • Pact 🆓 - Contract testing framework for microservices and API consumers.
  • MSW (Mock Service Worker) 🆓 - API mocking library for browser and Node.js, ideal for frontend AI testing.
  • Hoverfly 🆓 - Lightweight service virtualization with proxy-based recording and replay.

Performance Testing with AI

AI-enhanced performance, load, and chaos testing.

  • k6 🆓💰 - Open-source load testing tool, increasingly paired with AI for scenario generation.
  • WebPageTest 🆓💰 - Performance testing with AI-suggested optimizations.
  • LoadView 💰 - Cloud-based load testing with AI-powered insights.
  • Akamas 💰 - AI-driven performance optimization and tuning.

AI for Accessibility Testing

AI-powered accessibility scanners and remediation tools.

  • Pa11y 🆓 - Open-source automated accessibility testing tool.
  • Axe DevTools 💰 - AI-powered accessibility scanner from Deque.
  • Evinced 💰 - AI-driven accessibility testing platform.
  • UserWay 🆓💰 - AI-powered WCAG compliance scanner.

API Testing with AI

AI features for API testing, schema generation, and contract validation.

  • Keploy 🆓 - Generates API tests and mocks from real traffic, with AI assertion generation.
  • Stepci 🆓💰 - API testing framework with AI-suggested assertions.
  • Postman 🆓💰 - AI-assisted API test generation and contract validation with Postbot assistant.

LLM and AI System Testing

Tools to test LLM applications themselves (security, robustness, hallucination).

  • Garak 🆓 - LLM vulnerability scanner from NVIDIA.
  • DeepTeam 🆓 - LLM red teaming for prompt injection, jailbreaks, and data leaks.
  • llm-security-scanner 🆓 - Red-team toolkit with OWASP LLM Top 10 alignment and Turkish payload library.
  • Giskard 🆓💰 - Testing framework for LLMs and ML models.
  • Guardrails AI 🆓💰 - Python framework for validating and structuring LLM outputs using composable validators covering toxicity, PII leakage, and hallucination detection.
  • NeMo Guardrails 🆓 - Open-source toolkit from NVIDIA for adding programmable guardrails to LLM-based conversational systems, preventing jailbreaks, topic drift, and unsafe outputs.
  • PyRIT 🆓 - Microsoft's Python Risk Identification Tool for generative AI.
  • LLM Guard 🆓 - Open source security toolkit from Protect AI with scanners for prompt injection, toxicity, secrets, and data leakage in LLM inputs and outputs.
  • LLMFuzzer 🆓 - Early open-source fuzzing framework for testing LLMs via their API integrations. No longer actively maintained (last commit early 2024) but still referenced in LLM security lists.
  • Lakera Guard 💰 - Real-time prompt injection and jailbreak detection.
  • Prompt Security 💰 - Runtime prompt injection and data leak protection platform.
  • WhyLabs 💰 - ML observability and LLM monitoring.
  • Confident AI 💰 - LLM testing platform built around DeepEval.

Browser Automation for AI Agents

Browser automation libraries designed for or commonly used by AI agents.

  • Browser Use 🆓 - Make websites accessible to AI agents.
  • Stagehand 🆓 - AI browser automation with predictable yet flexible APIs.
  • Skyvern 🆓💰 - Automate browser-based workflows using LLMs and computer vision.
  • Steel Browser 🆓 - Open-source browser API for AI agents.
  • Patchright 🆓 - Patched Playwright for stealth automation.
  • Lightpanda 🆓 - Headless browser written in Zig, optimized for AI agents and scraping workloads.
  • AgentQL 🆓💰 - Natural language query language for AI agents to interact with and extract structured data from web pages, with self-healing selectors that integrate with Playwright.
  • Browserbase 💰 - Cloud browser infrastructure with natural language automation.

Articles and Talks

Essential reading on AI in software testing.

Courses and Tutorials

Learning resources for AI-powered testing.

Newsletters and Communities

Benchmarks and Datasets

  • BenchClaw 🆓 - Multi-dimension AI benchmark with 17-judge evaluation tribunal for scientific paper generation. Evaluates IMRaD structure, citation quality, methodological rigor, and reproducibility across 10 dimensions with uncertainty quantification and P2P verification.
  • HELM 🆓 - Stanford CRFM's open-source Python framework for holistic, reproducible, and transparent evaluation of LLMs and multimodal models across dozens of scenarios covering accuracy, robustness, efficiency, bias, and safety.
  • SWE-bench - Benchmark for evaluating LLMs on real software engineering tasks, including test fixes.
  • HumanEval - Evaluating large language models trained on code.
  • WebArena - Self-hostable web environment for building and evaluating autonomous agents on realistic, multi-site tasks.
  • OSWorld 🆓 - NeurIPS 2024 benchmark for evaluating multimodal AI agents on open-ended tasks in real computer environments, supporting VMware, Docker, and AWS virtualization.
  • tau-bench - Benchmark for evaluating tool-using language agents through dynamic conversations with simulated users and domain-specific APIs.
  • AgentBench 🆓 - ICLR 2024 benchmark for evaluating LLMs as autonomous agents across eight environments including OS, database, web browsing, and game tasks.

Contributing

Contributions are welcome. Please read the contribution guidelines before opening a pull request.