More screenshots
Works with the AI Agents you already use
Reviewer Calibration Kit
Build and score evaluation suites to measure recall, false positives, and verdict accuracy for LLM-as-a-judge agents.
$15
Reviewer Calibration Kit
Example session with this skill installed
I have a code reviewer agent. I need to know if it's actually catching security flaws or just guessing based on common patterns. How do I start a calibration suite?
- Read your context and instructions
- Compiled the reviewer calibration kit
I will initialize a new calibration suite for your code reviewer agent. This creates the necessary structure to define your security rules and author test cases.
Reviewer Calibration Suite: Security-Audit
The following directory structure has been scaffolded in ./security-audit/. You should now edit rulebook.md to define your security requirements and suite.json to assign severities.
security-audit/
├── rulebook.md # Define your "No hardcoded secrets" or "SQLi" rules here
├── suite.json # Map Rule IDs (e.g., SEC01) to blocker/major/minor
├── cases/ # Folder for your test material (K01, K02...)
│ └── .gitkeep
├── keys/ # Hidden ground truth for the scorer
│ └── .gitkeep
└── templates/
└── case-design.md # Map out your Planted vs. Twin strategy
Next steps
- Define Rules: Open
security-audit/rulebook.mdand write 3-5 specific, measurable security rules.
Design Cases: Use the templates/case-design.md to plan 5 planted defects (e.g., a real API key) and 5 twins (e.g., an environment variable reference).
3.
Lint Suite: Run python scripts/calibrate.py lint security-audit to ensure your configuration is valid before running the agent.
Connects securely to your tools. The creator never sees your data.
About this skill
Reviewer agents that "seem to work" often miss critical defects or hallucinate problems that don't exist. This skill provides a systematic framework to quantify the reliability of LLM-as-a-judge agents using a rigorous calibration process.
What it does
- Scaffold evaluation suites by generating structured folders for rules, test cases, and hidden keys.
- Design adversarial test cases including planted defects, near-miss clean twins, and prompt injection attempts.
- Automate judge execution across multiple runs to measure consistency and calculate statistical performance.
- Generate performance reports with metrics for recall, false positives, verdict accuracy, and invalid output rates.
- Benchmark agent tiers by comparing high-performing models against weaker ones to validate suite difficulty.
How it works
- Initialize the suite with
calibrate.py initto define your rulebook and severity levels (blocker, major, minor). - Author cases using the
case-design.mdtemplate, pairing every defect with a near-identical clean twin to test for true understanding. - Execute the judge via CLI with
calibrate.py run, which sends cases to your agent and saves structured JSON responses. - Calculate scores using
calibrate.py scoreto generate Markdown reports comparing agent findings against hidden keys.
Frameworks & tools
This kit uses a standalone Python 3.10+ CLI tool (calibrate.py). It is compatible with any LLM judge accessible via shell commands (e.g., Claude Code, local scripts, or API wrappers).
Why this beats prompting it yourself
Evaluating a judge with another LLM just shifts the trust problem. This tool uses a deterministic, dependency-free Python script to score agents against ground-truth keys. It forces you to define rules and severities upfront, ensuring your "calibration" is a measurable audit rather than a vibes-based check.
Use cases
- Validating an automated PR reviewer agent before deploying it to a production repo.
- Calibrating a document compliance grader to ensure it doesn't hallucinate legal violations.
- Benchmarking different LLM providers to find the cheapest model that maintains 1.0 blocker recall.
- Identifying "severity drift" where an agent correctly finds a flaw but wrongly downgrades its importance.
Known limitations
The scorer requires agents to output valid JSON in a specific schema. Suites with fewer than 20 cases are flagged as statistically underpowered. Requires a CLI-accessible judge.
Requires Python 3.10+. Agent must support JSON output. Suites with <20 cases trigger statistical significance warnings. No built-in LLM providers; requires user-supplied CLI command.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 11 days ago
Requires