Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    AI Agent Regression Testing & Release Gate.

    1

    Deterministic regression testing for AI prompts and agentic tool-use workflows using captured JSONL run records.

    $10

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    AI Agent Regression Testing & Release Gate.

    AI Agent Regression Testing & Release Gate.

    Example session with this skill installed

    Create a test suite for a weather agent that must call the get_weather tool with a city name and verify the final response contains a temperature.

    • Read your context and instructions
    • Compiled the ai agent regression

    I have generated a deterministic test suite for the weather agent. This suite validates both the tool execution and the presence of data in the final response.

    Weather Agent Test Suite

    {
      "suite_name": "Weather Agent Verification",
      "test_cases": [
        {
          "case_id": "weather-lookup-san-francisco",
          "title": "Basic Weather Query",
          "input": "What is the weather in San Francisco?",
          "assertions": [
            {
              "type": "tool_call_exists",
              "tool_name": "get_weather",
              "expected_arguments": { "city": "San Francisco" }
            },
            {
              "type": "text_contains",
              "pattern": "degrees"
            },
            {
              "type": "json_schema",
              "path": "metadata",
              "schema": { "type": "object", "required": ["model", "run_id"] }
            }
          ],
          "tags": ["smoke-test", "core-functionality"]
        }
      ]
    }
    

    Next steps

    • Capture a run of your weather agent into a run-record.jsonl file.
    • Execute the validation script: python scripts/prompt_suite.py weather_suite.json --run run-record.jsonl.
    • Review the generated eval.md report to confirm the tool arguments were mapped correctly by the model.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Regression in AI behavior often goes unnoticed until a user reports it. Prompt changes can break JSON structures, skip tool calls, or fail safety checks without warning. This skill provides a deterministic framework for validating agentic workflows against captured run records.

    What it does

    • Deterministic validation compares captured agent outputs and tool traces against a set of predefined assertions.
    • Tool call auditing verifies that specific tools were invoked in the correct order with the expected arguments.
    • Regression monitoring flags when model responses drift from a reference snapshot or fail structural requirements.
    • Safety enforcement applies tagged test suites to ensure privacy and safety constraints are maintained across model updates.
    • Standardized reporting generates Markdown and JSON evaluation reports for CI/CD integration or human review.

    How it works

    1. Define scenarios by creating a JSON test suite containing inputs, expected tool calls, and text assertions.
    2. Capture run records from your application as JSONL files containing the actual model outputs and metadata.
    3. Execute the runner using the provided script to evaluate the captured records against your test suite.
    4. Analyze reports to identify failed assertions, missing tool calls, or unexpected schema changes.

    Frameworks & tools

    This skill is provider-neutral and works with any LLM stack that can export JSON/JSONL traces. It uses a Python-based execution script for local or CI/CD validation of structured data.

    Why this beats prompting it yourself

    Manual inspection of agent traces is slow and prone to human error. This skill replaces vague "vibes-based" testing with strict, repeatable assertions that block broken builds before they reach production.

    Use cases

    • Verify that a customer support agent calls the refund_policy tool when asked about returns.
    • Ensure a data extraction prompt always returns valid JSON matching a specific schema.
    • Confirm that a model update hasn't introduced regressions in a complex multi-step reasoning chain.

    Known limitations

    This runner is deterministic and does not perform semantic or LLM-based evaluation. It requires external capture of run records as it does not call model APIs directly.

    Does not perform semantic/LLM-based evaluation. Does not call model APIs directly; requires pre-captured JSON/JSONL run records.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    Requires

    Python

    Permissions

    Terminal / Shell
    Listed1 month ago

    What's inside

    Frequently Asked Questions

    This skill prevents 'vibes-based' deployment by providing a deterministic way to verify that prompt changes don't break JSON structures, tool-call sequences, or safety constraints.

    The skill is provider-neutral and compatible with any AI framework (like LangChain, AutoGPT, or custom stacks) as long as you can export your agent's execution traces into a standard JSONL format.

    You will receive a Python-based execution script, a set of standardized JSON schema templates for defining your test suites, and documentation on integrating the runner into your CI/CD pipeline.

    No, this runner is designed for privacy and speed; it performs local deterministic validation against your captured records and does not require making additional, costly calls to LLM APIs.

    Setup involves installing the Python requirements, defining your 'gold standard' test scenarios in a JSON file, and pointing the runner to your captured JSONL run records.

    Currently, the runner focuses on deterministic assertions (exact matches, schema validation, and tool order); it does not perform semantic 'fuzzy' logic or LLM-based grading.

    New in Testing & Debugging