- Home
- Skills
- Testing & Debugging
- Flaky Test Hunter: reproduce, classify, fix
Works with the AI Agents you already use
Flaky Test Hunter: reproduce, classify, fix
The usual responses make things worse: hit "re-run", add a `sleep`, turn on suite-wide retries, or quietly delete the test.
$9
Flaky Test Hunter: reproduce, classify, fix
Example session with this skill installed
CI fails on tests/test_invoice.py::test_due_date maybe once a week and passes on re-run. Find out why and fix it.
- Read your context and instructions
- Compiled the flaky test hunter
Example session (illustrative).
1. Reproduce in isolation.
pytest tests/test_invoice.py::test_due_date --count=300 -x -p no:randomly gave 0/300 failures. The test does not fail on its own.
2. Reproduce in shuffled suite order.
pytest tests/ with pytest-randomly installed, run 20 times, failed twice: seeds 48213 and 9917. pytest tests/ --randomly-seed=48213 fails every time, so this is order dependence.
3. Find the polluter.
detect-test-pollution --failing-test tests/test_invoice.py::test_due_date --tests tests/ narrowed it to tests/test_settings.py::test_override_timezone.
4. Root cause.
test_override_timezone sets os.environ["TZ"] = "Asia/Tokyo" and calls time.tzset(), and never restores either. test_due_date computes "end of today" with naive datetime.now(), so after the polluter runs the date shifts by a day.
5. Fix. Two changes:
test_override_timezonenow usesmonkeypatch.setenv("TZ", ...)plus a fixture that callstime.tzset()again on teardown.test_due_datefreezes time with@freeze_time("2025-01-31 23:30:00")and asserts the month-end rollover explicitly.
Side finding: invoice.due_date() uses naive local time in production code. Flagged as a real bug, since invoices issued near midnight UTC get the wrong due date.
6. Verify.
pytest tests/ --randomly-seed=48213 passes. 30 shuffled suite runs gave 0/30 failures. Isolated: 0/300.
Test: tests/test_invoice.py::test_due_date
Status: FIXED
Repro: pytest tests/ --randomly-seed=48213
Before: 2/20 shuffled suites failed After: 0/30
Class: order dependence + time/timezone
Cause: test_override_timezone leaked TZ; test relied on wall clock
Fix: monkeypatch + tzset teardown; freeze_time in test_due_date
Follow-up: production bug, due_date() uses naive local time (issue drafted)
Connects securely to your tools. The creator never sees your data.
About this skill
The problem. Flaky tests erode trust in CI. The usual responses make things worse: hit "re-run", add a sleep, turn on suite-wide retries, or quietly delete the test. Coding agents are especially prone to this. They see one green run and declare victory.
What the agent does with this skill.
- Reproduces with real numbers. Uses the rule of three (about
3/pclean runs for 95% confidence) and the exact repeat commands for each stack:pytest-repeat --count, serial Jest loops,go test -count=N -race -cpu 1,2,8, and cargo test loops. - Diagnoses from the failure pattern. Checks whether the test fails alone, only in suite order, only in parallel, or only in CI. Each pattern points to a different cause class.
- Runs in random order and keeps the seed. Uses pytest-randomly,
jest --randomize --showSeed,vitest --sequence.shuffle,go test -shuffle=on, and nightly libtest--shuffle, then replays the failing seed. For order-dependent Python tests it finds the polluting test withdetect-test-pollution. - Applies the fix for the cause class. For example: freeze or inject the clock, seed RNGs and log the seed, wait on events instead of sleeping, use per-test temp dirs, bind port 0, restore env vars, reset state in teardown, and catch leaks with
--detectOpenHandlesor goleak. - Quarantines only with a paper trail. The test is skipped with a reason and an issue link, it stays visible, and its owner is named. It never deletes tests or retries until green.
- Writes a short report per test. Each report has the repro command, the before and after failure counts, the cause class, the fix, and any follow-up.
Who it's for. Teams whose CI is "red, re-run, green", maintainers triaging intermittent failures, and anyone letting coding agents touch test suites.
Requirements. Any agent that supports SKILL.md (Claude Code, Cursor, Codex CLI, and others) with shell access to run the project's tests. Optional helpers the agent may suggest installing: pytest-repeat, pytest-randomly, detect-test-pollution, goleak, serial_test. Rust order shuffling needs a nightly toolchain. It works on its own and pairs well with AgentHarness's no-network sandbox.
Supports JS/TS, Python, Go, and Rust only. Extremely rare flakes may require execution times beyond standard agent timeouts to hit statistical significance.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 17 days ago
Requires