- Home
- Skills
- Testing & Debugging
- Eval Quality Gate — Make Agent Eval Scores Mean Something
Works with the AI Agents you already use
Eval Quality Gate — Make Agent Eval Scores Mean Something
Audit LLM-agent eval suites for leakage, flakiness, and unfalsifiable judges to ensure scores are trustworthy.
$13
Eval Quality Gate — Make Agent Eval Scores Mean Something
Example session with this skill installed
Audit the distribute-weights task in the current directory. The judge is in judges/task-01.test.mjs. Check for leakage and falsifiability.
- Read your context and instructions
- Compiled the eval quality gate
The audit for the distribute-weights task is complete. I performed three runs of the test suite and attempted manual mutations of the implementation to verify the judge's sensitivity.
Eval audit: distribute-weights
| # | Criterion | Verdict | Falsification attempted | Evidence |
|---|---|---|---|---|
| 1 | Falsifiable judge | PASS | Submitted solution with sum mismatch | judges/task-01.test.mjs:22 |
| 2 | Train/eval leakage | FAIL | Grep found solution in fixtures/ref.txt | tasks/distribute.json:12 |
| 3 | Missing edge cases | REVISE | No check for zero-weight inputs | tasks/distribute.json:08 |
| 4 | Non-determinism | PASS | 3x runs returned identical results | CLI output |
| 5 | Tautological checks | PASS | Mutation of sign logic caused failure | judges/task-01.test.mjs:45 |
Patches for REVISE items
The prompt states "if the sum of weights is 0, distribute as evenly as possible," but the judge only tests positive integers.
Add the following check to judges/task-01.test.mjs:
test('handles zero total weight', () => {
const result = distribute([0, 0, 0], 100);
assert.deepStrictEqual(result, [33, 33, 34]);
});
Overall: FAIL
The suite's current pass rate is invalid due to
Criterion 2 (Leakage). The agent can recover the distribution logic directly from the reference fixture without reasoning. The judge also fails to enforce the zero-weight edge case defined in the prompt.
Next steps
- Remove the corrected code pattern from
fixtures/ref.txtand replace it with the broken version. - Apply the missing zero-weight test case to the judge file.
- Re-run the audit to move the overall verdict to PASS.
Connects securely to your tools. The creator never sees your data.
About this skill
Testing an agent on a suite it has already seen or using judges that cannot fail leads to inflated pass rates. This skill audits LLM-agent eval suites to ensure their scores represent actual reasoning capacity rather than leakage or flakiness.
What it does
- Audit judge rubrics for falsifiability by attempting to pass deliberately incorrect solutions.
- Detect data leakage where the solution exists verbatim in prompts, fixtures, or context.
- Identify missing edge cases by mapping prompt constraints (e.g., zero, negative, malformed inputs) to judge coverage.
- Test for non-determinism by running tasks multiple times to catch wall-clock assertions or unseeded randomness.
- Expose tautological checks that assert types instead of values or use the implementation as its own oracle.
How it works
- Analyze task files for required fields including
id,timeoutMs,fixtures, and explicit prompt restrictions. - Execute falsification attempts for each of the 5 core criteria, such as mutating implementation code to see if the judge still passes.
- Run the suite at least three times to verify result stability and identify flakiness.
- Generate a verdict report with PASS, REVISE, or FAIL statuses backed by file and line-level evidence.
Frameworks & tools
This skill is designed for JSON-based eval task suites and JavaScript/TypeScript test runners. It requires access to task definitions, fixtures, and the ability to execute the judge commands locally.
Why this beats prompting it yourself
Standard prompts often take eval rubrics at face value without testing their failure modes. This skill uses a systematic falsification framework to prove a judge can actually fail before it is allowed to pass.
Use cases
- Verify coding agent benchmarks before publishing or trusting performance claims.
- Debug "too good to be true" scores in internal RAG or agentic workflows.
- Harden eval suites by identifying prompt rules that lack corresponding judge assertions.
Known limitations
Does not ship automation scripts. Requires the user to provide a suite following the id/fixtures/judge file format. Only audits logic, not the statistical significance of the sample size.
Requires eval suites to follow a specific JSON task format. Does not perform network calls. Reasoning-heavy audit that requires local execution of the test runner.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 20 days ago