Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    AI Eval Test Suite Quality Gate Skill Pro

    1

    Audit LLM evaluations and test suites for gameable rubrics, data leakage, and reliability issues.

    $14

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    AI Eval Test Suite Quality Gate Skill Pro

    AI Eval Test Suite Quality Gate Skill Pro

    Example session with this skill installed

    Audit my new RAG eval suite. I've attached the judge_rubric.md, the test_cases.json, and the eval_script.py. We are seeing high scores but frequent hallucinations in prod.

    • Read your context and instructions
    • Compiled the ai eval test

    I have completed the audit of your RAG evaluation suite. While the scoring logic is sound, a critical loophole in the judge rubric and significant edge-case gaps render the current 94% pass rate untrustworthy.

    Overall Verdict: REVISE

    Audit Summary

    IDSeverityCategoryEvidenceConfidence
    FIND-01CriticalJudge Rubricjudge_rubric.md: Lines 12-18High
    FIND-02HighEdge-Case Coveragetest_cases.json: Missing null/empty contextHigh
    FIND-03MediumFlakinesseval_script.py: Temperature 0.7 on JudgeMedium

    Findings & Remediation

    [FIND-01] Critical — Verbosity Bias in Judge Rubric

    Observation: The rubric rewards "comprehensive and detailed" answers without a penalty for irrelevant information.

    Risk: The model can achieve a '5/5' score by hallucinating extra details as long as it maintains a professional tone, explaining the gap between eval scores and production failures.

    Recommended fix: Add a conciseness constraint and a "Negative Constraints" section to the rubric that penalizes information not present in the source context.

    [FIND-02] High — Missing Adversarial Contexts

    • Observation: The dataset only contains samples where the answer exists in the context.
    • Risk: The system's ability to "refuse to answer" when context is missing or irrelevant is completely untested.

    Recommended fix: Add 15-20 "No-Answer" test cases where the retrieval step is mocked to return empty or unrelated documents.

    Next steps

    1. Update judge_rubric.md with the provided conciseness constraints.
    2. Inject "Unanswerable" cases into test_cases.json to measure refusal accuracy.
    3. Pin the Judge LLM temperature to 0 in eval_script.py to reduce scoring variance.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Your LLM evaluation suite might be lying to you. High scores often hide gameable rubrics, data leakage, and brittle metrics that fail in production. This skill audits your AI test pipelines to ensure your quality gates are actually trustworthy.

    What it does

    • Judge Rubric Audit identifies vague scoring criteria, position bias, and loopholes in LLM-as-a-judge prompts.
    • Leakage Detection finds instances where ground-truth labels or reference answers have contaminated the model context or training splits.
    • Edge-Case Assessment maps your test cases against adversarial inputs, boundary values, and failure modes like timeouts or partial outputs.
    • Reliability Review flags nondeterminism caused by unpinned model versions, lack of seeds, or race conditions in the test harness.
    • Scoring Verification validates aggregation logic to ensure averages don't mask critical failures in specific data slices.

    How it works

    1. Inventory: Provide your system prompts, judge rubrics, evaluation datasets, and test scripts.
    2. Analysis: The agent audits the artifacts against seven dimensions including validity, flakiness, and reproducibility.
    3. Verdict: Receive a PASS, REVISE, or FAIL verdict based on evidence-backed findings.
    4. Remediation: Follow the generated plan to fix critical leakage or gameable scoring logic.

    Frameworks & tools

    Works with any evaluation stack including LangSmith, Promptfoo, DeepEval, or custom Python/Node.js test harnesses. It analyzes YAML, JSON, and code-based test definitions.

    Why this beats prompting it yourself

    Generic prompts often miss subtle data contamination or judge biases that skew results. This skill applies a systematic audit framework that separates observed defects from hypothetical risks and provides a formal severity/confidence matrix.

    Use cases

    • Auditing a new RAG evaluation pipeline before a production release.
    • Investigating why model benchmarks don't match real-world user feedback.
    • Strengthening judge rubrics to prevent models from "gaming" high scores through verbosity.
    • Verifying dataset split hygiene to prevent over-optimistic performance metrics.

    Known limitations

    Does not execute live code or perform network-level security testing. Review is limited to the artifacts and datasets explicitly provided or accessible in the environment.

    Limited to artifact analysis; does not execute code or live model calls. Requires access to rubrics and datasets for high-confidence verdicts. Scope is restricted to provided files.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 20 days ago

    Permissions

    No special permissions declared or detected

    Listed20 days ago

    What's inside

    Frequently Asked Questions

    This skill audits your entire testing infrastructure to identify hidden biases, data leakage, and 'gameable' rubrics that often lead to inflated performance scores and production failures.

    The skill is framework-agnostic and works with any stack—including LangSmith, Promptfoo, or custom Python/Node scripts—by analyzing your exported rubrics, test datasets, and system prompts.

    Setup is straightforward: you provide your existing test artifacts (YAML, JSON, or code) and the agent performs an automated audit against seven key reliability dimensions.

    No, the skill performs a static and logic-based audit of your evaluation artifacts and datasets; it does not require live execution or access to your production environment.

    You receive a comprehensive audit report with a PASS/FAIL verdict, a severity/confidence matrix for detected risks, and a specific remediation plan to harden your test suite.

    It is highly effective at identifying 'verbosity bias' and vague scoring criteria that allow models to score high simply by being more wordy rather than more accurate.

    New in LLM Ops