More screenshots
Works with the AI Agents you already use
Judge Sanity Check — Is Your LLM Judge Stable?
Reads your paired verdicts and reports chance-corrected agreement, position flips, verbosity bias, self-preference and sample size — before a judge score gates a release.
Free
Judge Sanity Check — Is Your LLM Judge Stable?
Example session with this skill installed
python3 scripts/judge_check.py judgments.jsonl --fail-on high
{"item_id":"i04","pair_id":"p04","order":"AB","judge_choice":"A","human_choice":"A",
"len_a":240,"len_b":120,"judge_model":"judge-x","a_model":"model-a","b_model":"model-b"}
- Read your context and instructions
- Compiled the judge sanity check
judge-sanity-check 80 judgments 80 labelled
agreement 38.8% kappa -0.065 (CI -0.231..0.111)
position flips 23.3% slot-A 52.5% verbosity 58.3% (n=12)
HIGH J2 chance-corrected agreement Cohen's kappa -0.07 is barely above chance
HIGH J3 position bias 23.3% of pairs changed verdict when swapped
HIGH J6 self-preference Judge picks own output 75% vs human 50%
FAIL findings at or above 'high'
Connects securely to your tools. The creator never sees your data.
About this skill
An LLM judge can give you a clean number every run and still be measuring the order you showed the candidates. This reads the verdicts you already have and tells you whether the score is stable enough to gate anything.
It computes chance-corrected agreement with a bootstrap confidence interval, not just raw agreement. It compares the same pair shown in both orders and reports how often the verdict flips. It checks whether the first slot wins more than half the decided pairs, whether the longer answer wins pairs humans called equal, and whether the judge favours its own model's output against the human rate. It reports the tie rate, the label distribution and the number of pairs behind every number.
- Cohen's kappa with a 95% bootstrap interval, plus the raw agreement and marginals
- Position flips and first-slot rate from pairs with both orders
- Verbosity bias measured on human-tied pairs, where length is the only difference
- Self-preference delta against the human baseline
- Sample-size and label-degeneracy checks, so a confident claim is not built on 20 rows
Runs on the standard library. JSONL in, Markdown and JSON out, exit 1 at a severity threshold for CI.
The shipped failing file is 80 verdicts with a judge near chance (kappa −0.07), 23% position flips, 75% self-preference and 58% longer-wins on human ties. The clean file is 60 verdicts with kappa 0.61, no flips and no length bias.
Good to know: agreement with humans is not validity — the humans can share the judge's blind spot, and a stable judge can be consistently wrong. Kappa depends on class marginals, so a low value can reflect a skewed sample as well as a bad judge. The bands used here are triage heuristics, not a published standard. Length and model fields are optional; missing fields are reported, never imputed. This audits verdicts; it cannot tell you whether the rubric measures quality.
Kappa depends on class marginals and may be low on skewed samples. Cannot verify rubric validity. Requires Python 3.10+.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 1 day ago
- Free with an Agensi account