flaky-detector

May 16, 2026 · View on GitHub

hero

flaky-detector

Run your test command N times. Find out which tests don't always agree with themselves.

License: MIT Python 3.8+ Claude Code Plugin Tests: 11 passing

TL;DR: /flaky-detector --cmd "pytest -v" --runs 10 → per-test flakiness %, sorted worst-first, ready to triage.

Writing

LinkedIn

Dev.to

Writing

Install (Claude Code)

git clone https://github.com/mturac/pluginpool-flaky-detector ~/.claude/plugins/flaky-detector

Restart Claude Code; the slash command /flaky-detector appears.

Writing

Quick start

python3 scripts/flaky.py --cmd "pytest -v" --runs 10 --format md
python3 scripts/flaky.py --cmd "go test ./..." --runs 20 --parallel 4 --out report.json
python3 scripts/flaky.py --cmd "jest --ci" --parser jest --runs 5

Tip: prefer pytest -v over pytest -q so every result lands on its own line. If you use -q, flaky-detector still picks up the tail FAILED path::test summary lines and exits non-zero with a warning rather than reporting a false green.

Writing

Flags

FlagDefaultDescription
--cmdrequiredThe test command (single line, no shell wrapping)
--runs10How many times to invoke the command
--parallel1Concurrent runs (only safe for parallel-clean suites)
--parserautopytest, jest, gotest, tap, or auto
--outnoneWrite JSON report to this path
--formatjsonjson or md

Writing

Supported parsers

ParserMatches
pytest`tests/foo.py::test_bar PASSED
jest / vitest✓ name, ✗ name, PASS file, FAIL file
gotest--- PASS:, --- FAIL:, --- SKIP:
tapok N - name, not ok N - name

Writing

Exit codes

CodeMeaning
0No flakies, no always-failing
1At least one test is flaky (0 < flakiness_pct < 100)
2At least one test is always-failing
3Zero tests parsed but the runner reported activity — re-run with -v

Writing

Example output (markdown)

# Flaky-detector report (10 runs)

- flaky: **2**  |  always-failing: **0**  |  always-passing: 47

| test | pass | fail | flakiness % |
|---|---|---|---|
| tests/test_payment.py::test_idempotency | 6 | 4 | 40.0 |
| tests/test_search.py::test_index_warmup | 8 | 2 | 20.0 |

Writing

Limitations

  • --parallel > 1 only works for parallel-safe suites; otherwise concurrent runs share state and lie.
  • Streaming stdout from very long suites is buffered — be patient on the first run.
  • The parser is tuned for default reporters. Custom plugins (pytest-rich, etc.) may need a tweak.

Writing

Examples

Step-by-step walkthroughs with real input fixtures and the helper's actual output live in examples/. Three or four scenarios per plugin — from the happy path to the edge cases the test suite guards.

Writing

Part of the pluginpool family

Ten focused Claude Code plugins for everyday productivity: commit-narrator · pr-storyteller · test-gap · deps-doctor · env-lint · secret-guard · standup-gen · todo-harvest · flaky-detector · changelog-forge

Writing

License

MIT — see LICENSE. Contributions welcome.