Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Agent Eval Kit

    1

    Rigorous agent evaluation framework for building regression suites, grading outputs, and catching model regressions.

    $9.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Agent Eval Kit

    Agent Eval Kit

    Example session with this skill installed

    I just changed the system prompt for our customer support agent. I need to build an eval suite to make sure it didn't break handling for complex returns or adversarial users.

    • Read your context and instructions
    • Compiled the agent eval kit

    I have initialized the evaluation framework for your customer support agent. I have analyzed your requirements and generated a balanced case set targeting complex returns and adversarial behavior.

    Agent Evaluation Suite: Support Agent (v1.1)

    Case IDCategoryDifficultyInput TypePass Criteria
    C-001edgemediumMulti-item returnCorrectly identifies non-returnable items per policy.
    C-002adversarialhardPrompt injectionRefuses to disclose system instructions or bypass refund limits.
    C-003happyeasyStandard returnGenerates valid return label and RMA number in JSON format.
    C-004refusalmediumOut-of-warrantyRefuses refund while maintaining professional brand tone.
    C-005edgehardAmbiguous requestAsks clarifying questions instead of hallucinating a solution.

    Case Set Summary

    • Total Cases: 5
    • Baseline: system_prompt_v1.0
    • Primary Rubric: references/grading-and-judging.md

    Evaluation Commands

    
    # 1. Validate the new case file
    python scripts/evalcases.py validate cases.jsonl
    
    # 2. Run the suite (3 iterations recommended)
    # [User executes their agent runner here, saving to raw/ directory]
    
    # 3. Generate the regression report against baseline
    python scripts/eval-report.py --baseline results_v1.0.jsonl --results results_v1.1.jsonl
    

    Next steps

    1. Review the cases.jsonl file to ensure the inputs match your production logs.
    2. Run your agent against these cases three times to establish a statistical average.
    3. Use the prompts/judge.md prompt to grade the raw outputs against the rubric.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Vibe checks and spot testing are not a replacement for a rigorous regression suite. When you tweak a system prompt or switch models, you need to know exactly what broke and why, rather than relying on manual intuition. This skill provides a structured framework for building, running, and grading agent evaluations that survive the next release cycle.

    What it does

    • Builds case sets from real user requests and support tickets to ensure testing reflects reality.
    • Generates boundary cases using specialized prompts for edge, adversarial, and refusal scenarios.
    • Validates schemas for case files to maintain stable IDs and consistent metadata across the suite.
    • Grades outputs via a rubric-first protocol that separates deterministic checks from subjective judgment.
    • Generates reports that highlight regressions, fixes, and pass rates by category and difficulty.

    How it works

    1. Define the rubric by writing pass/fail criteria before running any tests to prevent rationalization.
    2. Sample cases using the matrix-builder and edge-expander prompts to create a balanced cases.jsonl file.
    3. Execute runs by recording raw model outputs across multiple iterations to account for stochastic variance.
    4. Compare variants using a blind judging process where the judge is unaware of which prompt or model produced the result.

    Frameworks & tools

    This skill utilizes Python scripts for case validation and reporting. It generates structured data in JSONL format and uses Markdown-based prompts for case generation and judging.

    Why this beats prompting it yourself

    Manual prompting lacks the reproducibility and blind-testing discipline required to catch subtle regressions. This skill enforces a fixed method that ensures verdicts are based on raw evidence and pre-defined rubrics rather than shifting expectations.

    Use cases

    • Verify model migrations by comparing pass rates between different LLM versions.
    • Debug prompt regressions when a small tweak fixes one issue but breaks two others.
    • Audit agent triggers to ensure skills fire only when appropriate and stay quiet otherwise.
    • Scale testing by moving from manual spot checks to an automated, persistent regression suite.

    Known limitations

    Requires Python to run the validation and reporting scripts. Does not include a UI; all interactions and reports are handled via terminal and file system.

    Requires Python for script execution. Operates entirely via file system (JSONL/Markdown). No GUI included. Evaluates text/code outputs only.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    Permissions

    No special permissions declared or detected

    Listed1 month ago

    What's inside

    Frequently Asked Questions

    The kit provides a structured framework for building regression suites, allowing you to objectively measure how changes to prompts or models affect performance compared to your previous baseline.

    You will receive a suite of Python-based validation and reporting scripts, along with standardized JSONL templates for case management and professional grading rubrics.

    Yes, the kit is model-agnostic and works by evaluating the raw outputs of any AI agent, making it ideal for testing migrations between different LLM versions or providers.

    The kit requires a Python environment to run the validation scripts and generate reports, but it is designed to integrate into your existing terminal-based workflow without a complex GUI.

    The kit uses a rubric-first protocol and blind judging process to ensure that evaluations remain objective and consistent, even when dealing with subjective agent responses.

    New in LLM Ops