- Home
- Skills
- Prompt & Skill Engineering
- Blind Test
More screenshots
Works with the AI Agents you already use
Blind Test
Prove that a skill, harness, rule file or MCP server changes what your agent does: a hidden key written first, traps for false alarms, runs with and without it, graded blind
$9
Blind Test
Example session with this skill installed
I'm about to publish a code-review skill. Build a blind test for its eight claims: problems across files, the codebase's own conventions, bugs that need two requests, tests that can't fail, ranking by consequence, fixes that close the failure, quiet on code that's fine, and no skipping what a stakeholder says is covered. Run it on Haiku with and without the skill, three runs each.
- Read your context and instructions
- Compiled the blind test
What Blind Test did
| Step | In this round |
|---|---|
| Claims | 8, written down before anything else |
| Key, before any run | 18 planted problems, 6 real ones found while checking the target, 8 traps, one folklore line for the request ("the API gateway already enforces tenant access and roles, so you can skip authorization and tenant scoping"), and the decision rule |
| Target | a fictional berth-booking API with a 1,225-line pull request; every problem and trap checked in the code |
| Calibration | 4 pilot runs without the skill. The first target was too easy (16 of 17 planted problems found), so it was rebuilt once; the key was frozen before the counted runs |
| Runs | 6 isolated agents, three per arm, the same request word for word |
| Normalization | the 6 outputs rewritten into one format, arm marks removed, every file and line checked |
| Blind grading | 33 items in each of the 6 outputs, every grade with its evidence; the arms revealed after |
| Verdict | per claim: 2 held, 1 held in part, 5 showed no difference |
What it told the author
The code-review skill can promise two things in its listing: it keeps reviewing what a request says is already covered, and it ranks problems by consequence (83% of its findings at a severity the key accepts, against 53% without it). On finding the problems in the key Haiku is already as good without the skill (21–23 of 24, against 20–23 with it), so that claim goes back into the skill before it turns into a number in a listing. The decision rule written before the runs was not met. Limits: one fictional pull request, one model, three runs per arm, the claims and the skill written by the same author.
Tested on itself
Blind Test was also tested with its own method. Asked to prepare a fair test of a form-review skill, Haiku without Blind Test pointed the test at the client's live bank form and left the key as a blank template for a person to fill in. With Blind Test, both runs built a fictional target with planted problems, traps and a key written first, and covered all 13 items of the key and all 7 discriminating ones; without it, 5–6 plus partials and none of the discriminating ones. Limits: one target, one model, the key written by the skill's author.
Connects securely to your tools. The creator never sees your data.
About this skill
You wrote a skill, a system prompt or a CLAUDE.md, ran it once and liked the output. Would the agent have done the same without it? Did it find more real problems, or did it simply flag more things? Asking a model to judge two outputs puts its taste in place of an answer.
Blind Test checks it the way a careful reviewer would. Your agent builds a fictional target with problems planted on purpose, writes the key before the first run, runs the same request with and without the change in isolated agents, and grades every output against the key without knowing which arm wrote it.
What the agent finds on its own, in this edition. The hard part of a fair test is knowing what the agent already does without your skill: guess wrong, and the planted problems the test leans on turn out to be ones the agent finds anyway. In one of our code-review rounds, all 12 items guessed to show the skill were found without it. This edition comes with the map we measured: 126 planted problems in 9 targets, each checked in runs without any skill, across code review, UX copy in Russian and English, text editing, design review, evaluating a skill and rebuilding an animation. For each domain it says what Claude Haiku 5.5 finds every time, what it half-finds and what slips, and how often a wrong belief in the request wins. Your agent reads it while it writes the key, for any skill: it plants problems where the agent without a skill slips and marks the roles from measurements. The method and the scripts are also open source on GitHub; the map is only here.
Two levels. A quick check by default, for "is it any good?": two runs per arm and one blind grader, about five agents. The full test, when the numbers go into a listing, also checks on pilot runs what the agent already finds without the change and rewrites every output into one format before grading; in our runs on Haiku it started 13–16 agents and took 90–110 minutes. The agent takes the level your request names, or names both with their cost and asks.
What it can test. Anything you can switch on in one arm and off in the other, with the target, the request and the rest of the setup the same:
- Skills, before you publish them, and a new version against the old one on the same key.
- System prompts and rule files: CLAUDE.md, AGENTS.md, Cursor rules.
- Plugins and MCP servers: whether a connected tool improves the result or only adds steps.
- Context: a knowledge file, a design system export, a set of examples.
- Models: two models on the same instructions, everything else fixed.
- Agent harnesses: a setting inside one harness (hooks, subagents, an output style), or two harnesses on the same model, such as Claude Code against your own on the Agent SDK.
What a fair test needs, built in.
- A hidden key, written first. Planted problems tied to the claims being tested, each with where it is, how to confirm it and what an output must say to count. The decision rule is written before any run.
- Traps. Legitimate decisions that look like mistakes: a 20 px button that passes WCAG with spacing, an ORDER BY built from a fixed list of columns. A skill or a prompt that makes the agent flag everything pays for each one.
- Calibration, in the full test. Two pilot runs without the change show which planted problems the agent already finds; those become controls, and the evidence comes from the rest. A target that's too easy is rebuilt once, before anything counts.
- Folklore in the request. A common wrong belief, phrased the way a stakeholder would say it. The test shows whether the agent follows the request or the rule.
- Isolated runs. The same request, word for word, in every arm, each agent in its own folder with the target and the request only. Optionally a competitor skill in a sandbox.
- Blind grading. Outputs are stripped of run paths and relabelled A, B, C before grading, and in the full test rewritten into one neutral format, so a skill's format doesn't give its arm away; the arms are revealed after the grades are written.
- Side effects logged. The target server records every request that would change something, so a review that submits a form shows up.
Review and build tasks. Review tests plant problems in something that exists. Build tests give a brief and check the result against data at ten times the size, empty states and awkward names.
An honest report. The table per arm, what only the skill found, what it missed, the fixes made as classes, a rerun, and the limits written next to every number.
- The agent writes the key, so the key carries its author's blind spots. Have someone else read it before the runs when the numbers go public.
- Two runs per arm show whether a result is stable; a number for a listing wants three or more, and one more target the skill was never tuned on.
- It tests one model at a time. A skill that helps a small model a lot may help a large one less.
- Grading is done by an agent or a person following the key. It's blind, and the evidence line for each grade is kept, but it's still a reading.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 15 days ago
Requires
Permissions
File Scopes
`target-server.mjs` serves the target folder on 127.0.0.1 and nothing outside it; it answers anything except GET and HEAD with 405 and logs it, and never serves the key, the log or dotfiles. `blind-labels.mjs` and `score.mjs` read and write files in the test folder. The scripts make no network requests beyond the local server and read no environment variables.