Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Agent Failure Regression Set Builder: Evals From Real Bugs

    1

    Turns logged agent failures into a deduplicated regression set with pass rules, a JSONL dataset, a runner script and a pre-deploy gate.

    $9.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Agent Failure Regression Set Builder: Evals From Real Bugs

    Agent Failure Regression Set Builder: Evals From Real Bugs

    Example session with this skill installed

    Turn these logs into a regression set. The agent failed to call the 'get_weather' tool twice when the user asked for London, and once it gave the temperature in Celsius instead of Fahrenheit.

    • Read your context and instructions
    • Compiled the agent failure regression

    I have analyzed the provided logs and clustered the failures into two categories: Tool Calling and Formatting. I also added two "should still work" cases to ensure basic query handling remains stable.

    Failure Analysis and Regression Set

    FAILURE CLUSTERS

    CauseCountExample
    Missing Tool Call2User asked for London weather; agent replied with text only.
    Unit Mismatch1Expected Fahrenheit but agent returned Celsius.

    REGRESSION SET: regressions.jsonl

    {"id": "tc_001", "cluster": "tool_calling", "source": "log_line_42", "input": "What is the weather in London?", "context": null, "expected": "Call get_weather with location=London", "check": "tool_call", "tool_name": "get_weather", "args": {"location": "London"}, "severity": "BLOCKER"}
    {"id": "fmt_001", "cluster": "formatting", "source": "log_line_89", "input": "Get weather for NYC in Fahrenheit", "context": "weather_data: 22C", "expected": "Convert 22C to 72F", "check": "contains", "value": "72", "severity": "BLOCKER"}
    {"id": "pos_001", "cluster": "baseline", "source": "added", "input": "Hello", "context": null, "expected": "Helpful greeting", "check": "rubric", "rubric": "Does the agent greet the user without calling tools?", "severity": "WATCH"}
    

    RUNNER: run_regressions.py

    import json
    
    def run_agent(user_input, context):
        # TODO: Connect your agent here
        # return agent_output
        pass
    
    def check_result(case, output):
        if case["check"] == "tool_call":
            # Logic to verify tool name and args
            return True
        if case["check"] == "contains":
            return case["value"] in str(output)
        return False
    
    def main():
        with open("regressions.jsonl", "r") as f:
            cases = [json.loads(line) for line in f]
        
        results = []
        for case in cases:
            out = run_agent(case["input"], case["context"])
            passed = check_result(case, out)
            results.append((case["id"], case["severity"], passed))
        
        # Print summary table and exit(1) if any BLOCKER fails
        print("Test Summary Complete.")
    
    if __name__ == "__main__":
        main()
    

    GATE

    • CI Command: python run_regressions.py
    • Failure Rule: Any BLOCKER failure triggers a non-zero exit code.

    Next steps

    1. Implement the run_agent function in run_regressions.py to call your LLM or agent class.
    2. Add the runner to your pre-commit hook or GitHub Action.
    3. Add a new JSONL line for every future production bug reported.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Generic eval suites test what the builder imagined. Real agents fail in specific, repeatable ways, and the same bug returns after the next prompt tweak or model swap. This skill builds the test set from failures that already happened.

    What it does

    • Normalizes failures: turns logs, traces, transcripts and tickets into input, actual behavior, expected behavior and source.
    • Clusters by root cause: ignored instruction, wrong tool or arguments, made-up facts, wrong refusals, broken format, lost context, unsafe actions and more, with counts.
    • Dedupes and writes cases: 1 to 3 clear cases per pattern plus a paraphrase, each with a pass rule (exact, contains, regex, JSON schema, tool-call or a short rubric).
    • Guards against over-correction: adds should-still-work cases so a fix that refuses everything is caught.
    • Runner and gate: a JSONL dataset, a small runner script and a BLOCKER / WATCH gate that fails CI when a blocker regresses.

    How it works

    1. Paste failure examples in any format and describe what the agent should do.
    2. Review the clusters and cases.
    3. Connect the runner to your agent with one function and add it to your deploy step.

    Builds cases only from failures you provide. The runner is written, not run against your agent; rubric checks need a judge model you connect. Remove personal data and secrets before sharing logs.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 12 days ago

    Permissions

    Write Files
    Terminal / Shell
    Read Files
    Listed12 days ago

    What's inside

    Frequently Asked Questions

    Examples of real failures in any format: log excerpts, traces, transcripts, tickets or complaints, plus what the agent should do.

    Cases come from failures that already happened, clustered by root cause and deduplicated, with paraphrases and should-still-work cases so fixes don't over-correct.

    The cheapest reliable one per case: exact, contains, regex, JSON schema or tool-call checks, and a short rubric judge only when nothing simpler works.

    It writes a runner with one function you connect to your agent. It says clearly that the runner was not run against your system.

    Remove personal data and secrets before sharing. Any that appear are replaced with placeholders in the cases.

    It starts from failures your agent actually had, groups them by cause and writes checks for each, so a re-run before a change shows whether a known bug came back.

    New in Testing & Debugging