- Home
- Skills
- Code Quality & Review
- Verified Best-of-N: Test-Gated Multi-Patch Selection
Works with the AI Agents you already use
Verified Best-of-N: Test-Gated Multi-Patch Selection
Run 3-5 independent fix attempts, drop any that fail your tests, pick the winner by blind A/B review, apply only on approval.
$19
Verified Best-of-N: Test-Gated Multi-Patch Selection
Example session with this skill installed
Use verified-best-of-n with 3 candidates. The retry helper in src/retry.js runs one attempt too few when maxAttempts is set. Fix it, keep the public API, don't weaken tests. Validate with npm test. Don't apply anything until I've seen the report.
- Read your context and instructions
- Compiled the verified best-of-n
Verified Best-of-3 result: candidate-2 selected. 2 of 3 candidates passed npm test; the baseline on the unmodified code failed 2 of 4 tests, confirming the tests reproduce the bug.
| Candidate | Tests | Score | Note |
|---|---|---|---|
| candidate-2 | passed | 1.00 | Minimal loop-bound fix; adds a regression test for maxAttempts: 1 |
| candidate-3 | passed | 0.00 | Also raised the default budget from 3 to 5 and loosened assert.equal(calls, 3) to assert.ok(calls >= 3) |
| candidate-1 | failed | — | Still ran only 2 of 3 attempts; 2 tests failed, so it never reached judging |
Why it won: two independent judges compared the survivors blind, once in each A/B order, and both preferred candidate-2 on all three criteria (spec, output, error handling). It changes only the loop bound and adds coverage, while candidate-3 changed a default the task never mentioned and weakened an existing assertion to get green. AB/BA agreement: 100%.
Winner patch: src/retry.js (+1/-3), test/retry.test.js (+14/-0), SHA-256 81b97548878c…
- // Intentional demo defect: maxAttempts should include the final allowed attempt.
- const attemptsToRun = Math.max(1, maxAttempts - 1);
- for (let attempt = 1; attempt <= attemptsToRun; attempt += 1) {
+ for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
Report: .git/verified-best-of/runs/20261024-163828-c6b7a8/report.md
Apply this patch to your working tree? Nothing is committed either way.
After you reply "apply it": the helper re-checked the repository, HEAD, a clean tree and the patch SHA-256, applied the patch (not staged, not committed) and re-ran npm test: 5 of 5 tests passed. Your original checkout was never touched before that point.
Connects securely to your tools. The creator never sees your data.
About this skill
One coding-agent attempt can look right and still be the worse fix. Verified Best-of-N makes your agent solve the same task several times in isolation, lets your own test suite reject the broken attempts, compares the survivors blind, and only then offers one patch for you to apply.
How it works
- Plan gate - your agent restates the task, candidate count (3 or 5), test command and setup command, and waits for your OK.
- Isolated candidates - each attempt runs in its own detached git worktree. Agents with subagents (such as Claude Code) run them in parallel; others run them one by one.
- Test gate - every patch is captured with a SHA-256 hash and run against your test command. Failing candidates never reach review.
- Blind A/B judging - survivors are compared as anonymous "Patch A / Patch B" on three criteria: specification adherence, output match, and error-signal handling. Each pair is judged in both orders to cancel position bias, and consistency is reported.
- Report - ranking, scores, the judges' reasons, excluded candidates with logs, and warnings (for example, a candidate that loosened or deleted tests).
- Apply gate - nothing touches your checkout until you approve. Before applying, the helper re-checks the repository,
HEAD, a clean tree and the patch hash, then re-runs your tests. It never commits.
What you get
SKILL.md- a step-by-step workflow any SKILL.md-compatible agent can followscripts/bestof.py- a dependency-free Python CLI (init, validate, blind judging files, rank, integrity-checked apply, cleanup)references/judging-rubric.md- the three criteria with evidence rules and a worked examplereferences/prompts.md- candidate and judge subagent prompts, plus a single-agent fallbackexamples/sample-report.md- a real Best-of-3 report
Safety by design
- No network access, no package installs, no API keys. The helper is pure Python standard library.
- No commits, pushes, stashes or resets. Worktrees are removed with
git worktree removeonly. - All run state lives in
.git/verified-best-of/, invisible togit status. - Flags candidates that touch, loosen or delete tests, so a "green" result cannot hide a weakened suite.
- Validation runs your own test command on your machine, so use it on repositories you trust.
Good fits
Off-by-one and boundary bugs, flaky logic, tricky edge cases, refactors that must not change behaviour, and any change where being right matters more than being fast.
Auto-detects npm / pnpm / yarn / bun test, pytest (uv, poetry or plain), cargo test, go test ./... and make test, or takes any command you give it.
Built by the author of dsh-llm-verifier, the open-source DeepSeek Harness plugin. This skill is a standalone, agent-agnostic rewrite of that workflow: no Harness, no Node build and no DeepSeek API key required.
Requires a clean git working tree with at least one commit and a runnable automated test command. Cost scales with the candidate count (roughly N times one attempt plus judging). Validation runs your repository's own code on your machine, so use it only on repositories you trust. In agents without subagents, candidates are produced one after another and judging is self-judged rather than blind; the report says so. Needs permission to run git commands that write under .git (git worktree). Tested on macOS and Linux, not on native Windows.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 23 days ago
Requires
Permissions
File Scopes
The helper script makes no network calls, installs nothing and reads no credentials. It passes your existing shell environment through to the test command you approve (adding CI=1 so runners stay non-interactive). Validation runs inside isolated git worktrees, then once more in your checkout only after you approve the winner.