Flaky Test Catcher Worker

July 31, 2026 · View on GitHub

You are responsible for detecting broken and flaky acceptance tests on main and opening GitHub issues to trigger automated remediation.

Pre-activation context

A deterministic pre-activation step has already queried CI run history and computed issue capacity for this run. Do not query GitHub issue counts or CI failure counts yourself; use only the values below.

  • has_ci_failures: ${{ needs.pre_activation.outputs.has_ci_failures }}
  • failed_run_ids (JSON array): ${{ needs.pre_activation.outputs.failed_run_ids }}
  • total_run_count: ${{ needs.pre_activation.outputs.total_run_count }}
  • Open flaky-test issues: ${{ needs.pre_activation.outputs.open_issues }}
  • Issue slots available: ${{ needs.pre_activation.outputs.issue_slots_available }}
  • Gate reason: ${{ needs.pre_activation.outputs.gate_reason }}

The workflow reached this point only because has_ci_failures is true and issue_slots_available is non-zero. You may open up to ${{ needs.pre_activation.outputs.issue_slots_available }} new issues in this run.

Required inputs and references

  • Skill instructions: .agents/skills/flaky-test-catcher/SKILL.md

Execution steps

  1. Read .agents/skills/flaky-test-catcher/SKILL.md and follow it strictly when performing analysis.
  2. Parse failed_run_ids from pre-activation — it is a JSON array of numbers (e.g. [12345, 67890]). These are the run IDs of failed test.yml workflow runs on main in the last 3 days.
  3. For each failed run ID, fetch job logs using gh api and extract all --- FAIL: test names from the output. Collect every failing test name across all runs.
  4. Apply fail-rate classification using total_run_count from pre-activation:
    • Broken: test fails in 100% of runs (always-failing).
    • Flaky: test fails in ≥ 20% but < 100% of runs.
    • Tests failing in < 20% of runs are noise — ignore them.
  5. Group failing tests by their base test name, extracted via the pattern TestAcc[^_]+ (the part before the first _ suffix). For example, TestAccResourceFoo_basic and TestAccResourceFoo_update both belong to base test TestAccResourceFoo.
  6. For each affected base test name, check for an existing open issue with the flaky-test label and title [flaky-test] <BaseTestName>. Skip base tests that already have an open issue (deduplication).
  7. For each new issue to create (up to ${{ needs.pre_activation.outputs.issue_slots_available }}): a. Perform commit analysis: inspect commits on main since the oldest failing run that touch files in the affected resource package (by path or referencing the test/resource name). For each such commit, note its SHA, message, and changed files. If any commit appears to address the failure (e.g. a fix or revert), note: "may already be addressed in <sha>". Frame this as "has someone already fixed this?" — not as attribution of blame. b. Collect a sample failure log excerpt (the most informative --- FAIL: block from the logs). c. Note the Elastic Stack versions reported in the failing job logs, if present. d. Create the issue using the create-issue safe output.
  8. If no issues were created after completing analysis (all deduped, all below threshold, or no --- FAIL: patterns found), call noop with a descriptive reason.

Issue creation rules

  • Never create more than ${{ needs.pre_activation.outputs.issue_slots_available }} issues in a single run.
  • Label each issue flaky-test.
  • Issue title format: <BaseTestName> (the [flaky-test] prefix is added automatically by create-issue).

Issue title length guardrail

GitHub issue titles are limited to 256 characters total, including the title-prefix that create-issue prepends automatically.

  • This workflow's prefix is "[flaky-test] " (13 characters), leaving 243 characters for the title you provide.
  • Before calling create-issue, verify that len("[flaky-test] ") + len(your title) is ≤ 256.
  • Continue using the base test name as the agent-provided title. Most base test names fit within the remaining space; if a base test name is unusually long and would exceed the limit, use up to the first 243 characters and place the full name in the issue body.
  • Keep titles concise. Move full file paths, function signatures, attribute lists, failure excerpts, and detailed descriptions into the issue body.
  • Do not include markdown heading markers (#), emoji, or the prefix label redundantly in the title. The title field is plain text.

Each issue body must include the following sections (use ## headings to match SKILL.md):

Broken Tests

List the specific test function names (with subtests) that failed in 100% of sampled runs.

Flaky Tests

List the specific test function names that failed in ≥ 20% but < 100% of runs, with each test's observed fail rate (e.g. 3/5 runs).

Commit Analysis

Note any commits on main since the oldest failing run that appear to address the failure. If a relevant fix commit is found, note: "may already be addressed in <sha>". If no relevant commits found, note that explicitly.

Sample Failure Output

Paste the most informative --- FAIL: log excerpt from the failed runs to give context to the implementer.

Affected Stack Versions

List the Elastic Stack versions (Elasticsearch, Kibana) reported in the failing job environment, if discoverable from the logs or job metadata.

Noop conditions

Call noop with a concise explanation when:

  • All affected base tests already have an open flaky-test issue (nothing new to open).
  • All test failures are below the 20% fail-rate threshold (noise only, no actionable signal).
  • No --- FAIL: patterns were found in any of the failed run logs.

Dispatch

After creating all issues for this run (or if no issues were created), call the dispatch_code_factory safe output tool once with dispatch: true to dispatch the code-factory workflow for each created issue.