Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    skill regression harness

    1

    Generate a cost-tiered regression testing harness to catch silent behavioral breakages after editing SKILL.md files.

    $16

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    skill regression harness

    skill regression harness

    Example session with this skill installed

    I just updated the Iron Rules in `skills/billing-agent.md` to be stricter about data privacy. Generate a regression harness to make sure I didn't break the existing refund trigger.

    • Read your context and instructions
    • Compiled the regression harness

    I have analyzed `skills/billing-agent.md` and designed a 35-case regression battery. This harness is configured to use Claude Haiku 4.5 for bulk testing and Claude Opus 5 for escalation and confirmation sampling. ## Regression Harness for Billing Agent **Estimated Run Cost:** - Bulk Pass (Haiku 4.5): $0.07 - Judging & Escalation (Opus 5): $0.12 - **Total Projected Cost:** ~$0.19 per run The following files have been generated in `tests/regression/billing-agent/`: 1. `config.yaml`: Defines model routing, API environment variables, and a `$0.50` safety ceiling. 2. `battery.yaml`: 15 trigger-positive cases (refunds), 10 trigger-negative cases (general billing), and 10 compliance cases mapped to your new privacy rules. 3. `run_battery.py`: Script to generate responses from the skill under test. 4. `score_responses.py`: Rubric-based grader that performs cost-tiered escalation. 5. `README.md`: Setup instructions and usage guide. ```yaml # config.yaml snippet models: bulk: "claude-3-haiku-20240307" frontier: "claude-3-opus-20240229" routing: confirmation_sample_rate: 0.1 escalate_all_failures: true costs: max_projected_cost_usd: 0.50 ``` **Next steps** 1. Review the test cases in `battery.yaml` to ensure they reflect your expected "strict" privacy behaviors. 2. Set your `ANTHROPIC_API_KEY` environment variable. 3. Run `python run_battery.py --estimate-only` to verify the setup before making live API calls.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Editing a shipped SKILL.md is like editing a live program without a compiler. A single wording change can silently break trigger accuracy or cause the agent to ignore its own Iron Rules. This skill solves the problem of expensive, manual re-testing by generating a portable, automated regression harness for your specific skill files. It provides the scripts you need to catch behavior drift before merging edits, using a cost-efficient model routing strategy that reserves frontier models only for critical verifications. ## What it does - **Battery generation** creates a structured set of 25 to 40 test cases covering triggers, compliance, and adversarial inputs. - **Cost-tiered routing** runs bulk tests on cheaper models like Haiku 4.5 and escalates failures to Opus 5 for confirmation. - **Rubric-based scoring** extracts specific rules from your skill text to judge outputs with traceable citations instead of vague scores. - **Regression triage** compares failures against your recent edits to distinguish between silent breakages and intended behavior changes. - **Budget enforcement** calculates and enforces API cost ceilings for both response generation and grading passes. ## How it works 1. **Analyze target skill** to identify all trigger surfaces and compliance rules within your SKILL.md. 2. **Design prompt battery** including positive/negative triggers and rubric-linked adversarial prompts. 3. **Generate harness code** including `config.yaml`, test scripts, and model routing logic for your local environment. 4. **Execute local scripts** to run the battery, score responses, and generate a detailed `scorecard.md` report. ## Frameworks & tools This skill generates Python-based scripts (`run_battery.py`, `score_responses.py`) that interface with the Anthropic API. It uses YAML for configuration and Markdown for reporting. ## Why this beats prompting it yourself Manual testing is inconsistent and expensive if you use frontier models for every check. This skill automates the "bulk-test, frontier-escalate" workflow, ensuring you only pay for high-end reasoning when a cheaper model flags a potential issue. ## Use cases - Validating a PR that modifies core agent instructions in a production skill. - Setting up CI/CD gates to prevent behavioral regressions in a skill portfolio. - Auditing third-party skills for rule compliance and adversarial robustness. - Reducing API spend during the maintenance phase of an agent's lifecycle. ## Known limitations This tool is for post-ship regression testing, not interactive first-draft authoring. It requires local Python execution and valid Anthropic API credentials.

    Requires local Python environment. Not for interactive first-draft authoring. Cost estimates are based on 2026-08-05 pricing data and may require manual updates if API prices change.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    Permissions

    Terminal / Shell
    Environment Variables
    Listed1 month ago

    What's inside

    Frequently Asked Questions

    It generates a regression test suite for one specific SKILL.md: a battery of prompts that checks the skill still triggers when it should (and stays quiet when it shouldn't), and still follows its own rules. Most cases run on a cheaper model (Haiku by default). Every case it marks as failed is re-checked on a frontier model before being reported, and a random sample of passing cases is re-checked too, so cheap-model mistakes in either direction get caught. It detects behavior changes after an edit; it does not prevent them.

    The skill instructions (SKILL.md), a battery design guide, a dated pricing and sourcing reference, and a templates folder with run_battery.py, score_responses.py, example config.yaml and battery.yaml files, and a README. When you use the skill, your agent fills in the battery and config for your specific skill from those templates. You review them, then run the scripts yourself.

    The scripts call the Anthropic API with your own API key (read from an environment variable) and charge your account for every run. Defaults are Claude Haiku 4.5 for the bulk pass and Claude Opus 5 for confirmation; both are set in config.yaml. You need a local Python environment with PyYAML and the anthropic package. The scripts print a projected cost first and refuse to exceed a spending ceiling without an explicit override.

    No. It is designed to be generated once and re-run after each edit, locally or in CI when a SKILL.md changes. The scripts and test cases are yours to keep. If you add or change a rule or trigger, update or add the matching test case so the battery keeps covering it.

    No. It is a regression test, not a security scanner. It includes a few "pressure" cases (for example a user pushing back or claiming authority) to check your skill's own rules still hold after an edit. All calls are text-only and no tools are given to the model. It does not scan for prompt-injection vulnerabilities in your agent.

    New in LLM Ops