Flaky Test Detector

Stop wasting 30 minutes per CI rerun on tests that pass locally but fail randomly in CI.

The Problem

Your CI pipeline red-lights every third build on a test that passes 90% of the time. You re-run it and it passes. Your team starts ignoring CI failures because "it's probably just flaky." Meanwhile, a real regression sneaks through because nobody trusts the test suite anymore. The root cause is always non-determinism — timing races, shared state, unmocked APIs, or environment drift — but finding which one takes tedious manual investigation every single time.

What You Get

• Automated flakiness confirmation — rerun loop scripts (50 iterations) that measure exact failure rates for pytest, Jest, Go, and Maven test suites
• Root cause classification engine — maps symptoms to 6 diagnosed categories: timing/race conditions, shared state, random data, external dependencies, environment differences, and teardown failures
• Isolation testing workflow — scripts to run tests alone vs. with neighbors, in shuffled order, and under parallel stress to pinpoint whether failures are order-dependent or resource-contended
• Framework-specific fix patterns — concrete before/after code for each root cause: replacing sleep() with poll-with-timeout, replacing class-level fixtures with per-test fixtures, seeding random generators, mocking external calls
• Quarantine system — generates @pytest.mark.flaky markers, describe.skip blocks, or @Disabled annotations with tracking issues, plus a FLAKY_TESTS.md registry with owner and quarantine date
• CI prevention pipeline — GitHub Actions YAML with pytest-rerunfailures, pytest-flakefinder, and flaky-detection steps that parse JUnit XML to flag tests with mixed pass/fail outcomes

Why This Beats Prompting It Yourself

When you ask an AI agent "fix my flaky test," it typically adds a longer sleep() timeout or wraps the assertion in a retry — treating symptoms instead of diagnosing the actual source of non-determinism. This skill forces a systematic 5-phase workflow: confirm → diagnose → fix → quarantine → prevent. Without it, agents gravitate toward the easiest-looking fix (more sleep, bigger timeout) rather than the correct one (eliminating the shared state or mocking the external dependency).

Example Session

You: Our test_payment_retry fails in CI about 1 in 5 runs but always passes locally. Here's the test file.

Agent: Detected 3 potential root causes from code scan: (1) time.sleep(2) on line 47 — timing-dependent assertion, (2) @pytest.fixture(scope='class') creates shared state between test_retry_success and test_retry_exhausted, (3) unmocked HTTP call to payments.stripe.com on line 62. Ran isolation test: passes alone, fails when paired with test_payment_webhook. Root cause: shared state — the class-level fixture retains payment state across tests. Fix: convert to function-scoped fixture. Applied fix, ran 50-iteration validation: 0/50 failures. Quarantined original with tracking issue #247 while verifying.

Use Cases

• Diagnosing why a test fails only on GitHub Actions Ubuntu runners but passes on macOS locally
• Systematically identifying which tests in a 500-test suite are flaky before a major release
• Adding a nightly flaky-test detection job that surfaces newly introduced non-determinism
• Setting up CI quarantine rules that block merging tests with >5% flakiness rate
• Investigating a test that fails when run with pytest-xdist -n auto but passes single-threaded

Known Limitations

Flaky tests caused by actual hardware differences (SSD vs. HDD timing, real network latency) cannot be fixed by code changes alone — they require environment standardization. This skill focuses on code-level non-determinism, not infrastructure-level flakiness.

---

Tags: testing ci-cd flaky-tests pytest jest debugging code-quality

Version: 1.0.0