- Home
- Skills
- Testing & Debugging
- Skill Bench: Catch Your Claude Skill's Bugs Before Buyers Do
More screenshots
Works with the AI Agents you already use
Skill Bench: Catch Your Claude Skill's Bugs Before Buyers Do
Find out whether your Claude skill really beats plain Claude: blind with/without runs, code checks, fix briefs and a strict no-regression bar.
$14.99
Skill Bench: Catch Your Claude Skill's Bugs Before Buyers Do
Example session with this skill installed
Bench my bill-splitter skill before I sell it. Three cases: an even split with tip, the same job in messy wording, and a trap where the tip % is missing. 2 runs with the skill, 1 without, graded blind.
- Read your context and instructions
- Compiled the bench
Skill Bench round 1: bill-splitter (invented demo skill, synthetic cases)
9 runs: 2 with the skill + 1 without, on 3 cases. Code checks first, then one blind grader per packet (no arm labels).
| With skill | Without skill | |
|---|---|---|
| Lines passed | 17 of 24 | 11 of 12 |
Verdict: the skill is WORSE than plain Claude on the core job.
Finding D-1 - skill defect
Every with-skill share is rounded down to a whole unit: 53 instead of 53.07, so the group pays 212 of 212.29. Seen in
4 of 4 with-skill runs on the two split cases,
0 of 2 without.
Cause (quoted from SKILL.md step 4): "Round each share down to the nearest whole unit of the currency so nobody has to deal with coins."
Blind grader, quoting the answer: "4 x EUR 53 = EUR 212, which is EUR 0.29 short of the grand total". Grade:
fail.
Test-set issue T-1
A code check failed an answer that only mentioned a service charge. The check is dropped and the grader judges that line.
Fix brief
Round each share to the cent and say who pays the leftover cent.
- Acceptance: 53.07 and 37.92 pass in 2 of 2 runs.
- Guard: the missing-tip trap must still ask for the tip %.
Round 2 (fixed skill)
- 24 of 24 lines pass.
bench.py regress: PASS, 0 regressions, 4 lines improved.- Grader spot-check: 4 of 4 grades re-read against the answers agree.
Connects securely to your tools. The creator never sees your data.
About this skill
A skill can pass one quick try and still fail on a repeat run, invent numbers, or make claims about itself that are not true. Skill Bench is a test method for catching that before your buyers do.
You point it at a skill. It helps you design a test set with real spread: different inputs, messy wording, traps where the skill's own rules must hold, and planted ground truth so answers are checked, not guessed. Nothing runs until you approve the set.
Every case runs with the skill and without it, several times. bench.py runs the code checks first (counts, lengths, numbers against a computed true value, forbidden terms, files produced). Then each answer goes to a grader in a shuffled, unlabelled packet, so the grader never knows which arm wrote it. Three grades: pass, pass with note, fail, each with a quoted reason.
Failures are classified as skill defect, test-set issue, harness artifact or grey area, and become a fix brief with the exact old and new wording, acceptance lines and a guard line for the fix's own risk. After you change the skill, the re-bench passes only if every acceptance line holds and nothing that passed before now fails.
It builds on Anthropic's skill-creator (its eval format, aggregator and review page) and adds the test design, blind grading, repeatability check and regression bar. It does not build skills or tune trigger descriptions; use skill-creator for that.
Compatibility
Best in Claude Code, or any Claude environment with sub-agents and Python 3.9+ (standard library only, no installs). Needs Anthropic's skill-creator skill for the benchmark summary and review page. In a plain claude.ai chat with no sub-agents it runs a labelled smoke test only, because one session cannot fairly write both arms. An optional API-batch route for large sets never spends without your go.
Permissions
- Read files: the skill you bench and your test set
- Write files:
bench.pywrites only to the bench workspace you point it at - Delete files: only
bench.py blind --force, which empties and rebuilds the packet folder you name - Run Python:
scripts/bench.pyon your own files; apythoncheck in your test set imports and runs a check module you write - Sub-agents: one per run and one per grading packet
- Network: none, except the optional
bench.py fresh --links, which checks that links in your own test set still load
Known limitations
- It is slower than skill-creator alone. On our own bench (v1.1, 6 synthetic cases) skill-creator took 37% less time and 7% fewer tokens per run.
- It does not build skills or tune trigger descriptions; it pairs with skill-creator.
- A bench only measures the cases you give it. In our own bench, two trap cases did not separate the arms: plain Claude passed them too.
- Blinding is partial: answers written with the skill can name
bench.py, which hints at the arm. - The grader is a model. Every grade quotes its evidence, and the method asks you to spot-check a few.
- Sub-agents can write outside their run folder (3 of 54 runs in our bench). Run it in a dedicated, empty folder.
- The number check reads a value written after its label in a sentence; values inside a table row go to the grader instead.
- Measured on one model family.
Claude is a trademark of Anthropic, PBC; this skill is not affiliated with or endorsed by Anthropic.
Requires skill-creator for full functionality. API cost estimates require manual verification if older than 30 days. Single-session runs lack a baseline comparison.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 month ago
Requires