Evidence-First Debugging

    1

    Evidence-first debugging: reproduce, measure flakiness, find the root cause, fix it with a regression test.

    Free

    7 usesSecurity scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Evidence-First Debugging

    Example session with this skill installed

    test_get_user_returns_record fails now and then in CI with a TypeError. Find out why and fix it properly.

    • Read your context and instructions
    • Compiled the evidence-first debugging

    Real session on a small Python repo. The script output is pasted as-is. ## Summary The test is flaky (6 of 20 runs fail). The cause is randomness in `get_user`, added in the most recent commit to the file. It's fixed at the cause, with a regression test, and now 0 of 20 runs fail. ## 1. Reproduce and measure ``` $ python3 scripts/repro.py -n 20 -- python3 -m unittest tests.test_cache python3 -m unittest tests.test_cache: 6/20 runs failed (FLAKY) ×6 FAILED (errors=1) hint: look for shared state, ordering, time, randomness or leftover files between runs ``` ## 2. Read the evidence ``` $ python3 scripts/test_summary.py -- python3 -m unittest tests.test_cache FAILED (exit 1) FAILED (errors=1) ERROR tests.test_cache.T.test_get_user_returns_record File "tests/test_cache.py", line 7, in test_get_user_returns_record TypeError: 'NoneType' object is not subscriptable ``` `get_user` sometimes returns `None`. ## 3. Check recent history ``` $ python3 scripts/recent_changes.py app/cache.py recent commits touching app/cache.py: f8174b0 2026-10-18 cache: skip refetch for speed a8e0514 2026-10-18 add user cache ``` ## 4. Root cause | Hypothesis | Evidence | Verdict | |---|---|---| | Shared state between tests | Only one test in the module, and it still fails 30% of runs | Ruled out | | Randomness in the cache | `app/cache.py` calls `random.random()` to decide whether to return the entry or `None` | Confirmed | ## 5. Fix at the cause, plus a regression test ```python _cache = {} def get_user(uid): if uid not in _cache: _cache[uid] = {"id": uid, "name": f"user{uid}"} return _cache[uid] ``` ```python def test_cached_user_is_always_returned(self): for _ in range(200): self.assertIsNotNone(get_user(2)) ``` ## 6. Verify ``` $ python3 scripts/repro.py -n 20 -- python3 -m unittest tests.test_cache python3 -m unittest tests.test_cache: 0/20 runs failed (never fails) ``` **Next steps:** run the full suite in CI, and check other "speed" commits for similar shortcuts.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Debugging by guessing leads to regression loops and masked symptoms. This skill enforces an evidence-first methodology to resolve crashes, hangs, and flaky tests by isolating the root cause before writing a single line of production code. It provides a structured workflow to reproduce failures, analyze error signatures, and verify fixes with statistical rigor. ## What it does - **Flakiness analysis** uses repetition scripts to determine if a failure is deterministic or intermittent. - **Error signatures** groups the distinct failures seen across repeated runs, so you can tell one bug from several. - **Change tracking** lists recent local commits that touched a file or a piece of text, to check whether a recent edit caused the failure. - **Hypothesis logging** maintains a record of attempted experiments and their results to prevent circular debugging. - **Regression testing** ensures every fix includes a test case that failed prior to the code change. ## How it works 1. **Reproduce** the failure using `repro.py` to establish a baseline failure rate and capture the exact stack trace. 2. **Isolate** the problem by bisecting input data or using `recent_changes.py` to identify relevant recent edits. 3. **Hypothesize** and test specific causes using prints or assertions, logging each result to confirm or rule out theories. 4. **Fix and Verify** by applying the root cause solution and running `test_summary.py` or high-iteration reproduction scripts. ## Frameworks & tools The skill utilizes Python 3 standard library scripts for reproduction and change analysis. It integrates with standard test runners like `pytest` and `unittest`. ## Why this beats prompting it yourself Standard prompts often jump to code fixes based on symptoms, which masks bugs rather than solving them. This skill enforces a strict sequence of reproduction and evidence gathering, ensuring you don't waste time on "lucky" fixes that break later. ## Use cases - Debugging intermittent CI failures that only happen 5% of the time. - Identifying the root cause of a silent hang in a data pipeline. - Checking whether a recent commit to a file introduced a regression. - Cleaning up complex error logs to find the first meaningful failure point. ## Known limitations Scripts require a Python 3 environment. It is designed for logic failures and crashes rather than visual UI inconsistencies or CSS debugging. Free and open source (Apache-2.0), by the author of [rosetta-js](https://github.com/Dreamstick9/rosetta-js), a terminal coding agent for open-weight models.

    Requires Python 3 standard library. Does not support visual/UI debugging. Effectiveness depends on the availability of a CLI command to trigger the failure.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    7 uses

    Times used to date

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    • Free with an Agensi account

    Requires

    python3
    git

    Permissions

    Terminal / Shell
    Environment Variables
    Read Files
    Write Files

    Shell runs the project tests and the helper scripts. Read/Write Files: the agent reads code and writes the fix plus a regression test. Environment variables are only passed through to the test command (with CI=1, NO_COLOR=1). No network access.

    Listed1 month ago

    What's inside

    Frequently Asked Questions

    This skill is specifically designed to eliminate "guess-and-check" debugging by enforcing a strict evidence-first workflow that isolates root causes through statistical repetition and log analysis.

    The SKILL.md workflow, three Python 3 standard-library scripts (repro.py, test_summary.py, recent_changes.py), a smoke test that checks the scripts, and the Apache-2.0 license. It's free.

    While optimized for Python-based environments and test runners, the methodology and analysis scripts can be applied to any codebase where failures can be triggered via a CLI or captured in logs.

    It is highly effective for intermittent issues; the skill uses automated repetition scripts to establish a baseline failure rate, helping you confirm when a flaky test is truly resolved rather than just hiding.

    The skill requires a Python 3 environment to run the included diagnostic tools and is designed to work with any AI agent capable of executing scripts and reading file outputs.

    This skill focuses on logic regressions, crashes, and performance hangs; it is not recommended for debugging visual UI layout issues or CSS styling inconsistencies.

    New in Testing & Debugging