Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Eval Quality Gate — Make Agent Eval Scores Mean Something

    1

    Audit LLM-agent eval suites for leakage, flakiness, and unfalsifiable judges to ensure scores are trustworthy.

    $13

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Eval Quality Gate — Make Agent Eval Scores Mean Something

    Eval Quality Gate — Make Agent Eval Scores Mean Something

    Example session with this skill installed

    Audit the distribute-weights task in the current directory. The judge is in judges/task-01.test.mjs. Check for leakage and falsifiability.

    • Read your context and instructions
    • Compiled the eval quality gate

    The audit for the distribute-weights task is complete. I performed three runs of the test suite and attempted manual mutations of the implementation to verify the judge's sensitivity.

    Eval audit: distribute-weights

    #CriterionVerdictFalsification attemptedEvidence
    1Falsifiable judgePASSSubmitted solution with sum mismatchjudges/task-01.test.mjs:22
    2Train/eval leakageFAILGrep found solution in fixtures/ref.txttasks/distribute.json:12
    3Missing edge casesREVISENo check for zero-weight inputstasks/distribute.json:08
    4Non-determinismPASS3x runs returned identical resultsCLI output
    5Tautological checksPASSMutation of sign logic caused failurejudges/task-01.test.mjs:45

    Patches for REVISE items

    The prompt states "if the sum of weights is 0, distribute as evenly as possible," but the judge only tests positive integers.
    Add the following check to judges/task-01.test.mjs:

    test('handles zero total weight', () => {
      const result = distribute([0, 0, 0], 100);
      assert.deepStrictEqual(result, [33, 33, 34]);
    });
    

    Overall: FAIL

    The suite's current pass rate is invalid due to

    Criterion 2 (Leakage). The agent can recover the distribution logic directly from the reference fixture without reasoning. The judge also fails to enforce the zero-weight edge case defined in the prompt.

    Next steps

    1. Remove the corrected code pattern from fixtures/ref.txt and replace it with the broken version.
    2. Apply the missing zero-weight test case to the judge file.
    3. Re-run the audit to move the overall verdict to PASS.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Testing an agent on a suite it has already seen or using judges that cannot fail leads to inflated pass rates. This skill audits LLM-agent eval suites to ensure their scores represent actual reasoning capacity rather than leakage or flakiness.

    What it does

    • Audit judge rubrics for falsifiability by attempting to pass deliberately incorrect solutions.
    • Detect data leakage where the solution exists verbatim in prompts, fixtures, or context.
    • Identify missing edge cases by mapping prompt constraints (e.g., zero, negative, malformed inputs) to judge coverage.
    • Test for non-determinism by running tasks multiple times to catch wall-clock assertions or unseeded randomness.
    • Expose tautological checks that assert types instead of values or use the implementation as its own oracle.

    How it works

    1. Analyze task files for required fields including id, timeoutMs, fixtures, and explicit prompt restrictions.
    2. Execute falsification attempts for each of the 5 core criteria, such as mutating implementation code to see if the judge still passes.
    3. Run the suite at least three times to verify result stability and identify flakiness.
    4. Generate a verdict report with PASS, REVISE, or FAIL statuses backed by file and line-level evidence.

    Frameworks & tools

    This skill is designed for JSON-based eval task suites and JavaScript/TypeScript test runners. It requires access to task definitions, fixtures, and the ability to execute the judge commands locally.

    Why this beats prompting it yourself

    Standard prompts often take eval rubrics at face value without testing their failure modes. This skill uses a systematic falsification framework to prove a judge can actually fail before it is allowed to pass.

    Use cases

    • Verify coding agent benchmarks before publishing or trusting performance claims.
    • Debug "too good to be true" scores in internal RAG or agentic workflows.
    • Harden eval suites by identifying prompt rules that lack corresponding judge assertions.

    Known limitations

    Does not ship automation scripts. Requires the user to provide a suite following the id/fixtures/judge file format. Only audits logic, not the statistical significance of the sample size.

    Requires eval suites to follow a specific JSON task format. Does not perform network calls. Reasoning-heavy audit that requires local execution of the test runner.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 20 days ago

    Permissions

    Terminal / Shell
    Read Files
    Listed20 days ago

    What's inside

    Frequently Asked Questions

    Audit LLM-agent eval suites for leakage, flakiness, and unfalsifiable judges to ensure scores are trustworthy.

    This skill uses the open SKILL.md standard. Use it through an MCP-compatible agent connected to Agensi, and check the compatibility information on this page for confirmed agents and required capabilities.

    Connect your AI agent to Agensi through MCP, then ask it to use Eval Quality Gate — Make Agent Eval Scores Mean Something on your task. Your agent loads the latest approved version each time. Paid skills require a purchase or active subscription. Skill files are not available for download; your licence does not allow you to save, copy or export any part of a skill.

    New in Testing & Debugging