Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    LLM Eval Harness

    1

    Evaluate LLM prompts and models with repeatable test suites, semantic checks, and statistical comparisons.

    $2.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    LLM Eval Harness

    LLM Eval Harness

    Example session with this skill installed

    Create a JSONL eval case for a customer support agent that must return a JSON object with a 'ticket_id' and cannot contain the word 'apologize'. Then show me how to run it.

    • Read your context and instructions
    • Compiled the llm eval harness

    I have drafted the evaluation case and the corresponding execution command. This setup ensures your agent follows the required JSON schema while adhering to your negative constraints.

    Evaluation Case Definition

    {"id": "support-001", "input": "Cancel my order #12345", "expected": "{\"ticket_id\": 12345}", "checks": ["json", "has_key:ticket_id", "not_contains:apologize", "max_words:50"], "tags": ["cancellation", "json-output"]}
    

    Execution Commands

    To run this evaluation against your local model script and generate a report:

    
    # 1. Run the evaluation
    python3 scripts/evals.py run cases.jsonl --cmd "python3 agent.py" -o results_v1.jsonl
    
    # 2. Score the results and generate a Markdown report
    python3 scripts/evals.py score cases.jsonl results_v1.jsonl --md report.md --min-pass 1.0
    
    # 3. View the summary
    cat report.md
    

    Next steps

    1. Add more variations of the "cancel" intent to the cases.jsonl file to increase test coverage.
    2. Integrate the score command into your GitHub Actions workflow to gate PRs on 100% pass rates.
    3. Use the compare command if you decide to swap the underlying model or adjust the system prompt.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Prompt tweaks and model swaps often break LLM features in ways unit tests miss. This skill provides a repeatable evaluation harness to quantify performance changes across datasets.

    What it does

    • Multi-check validation applies exact match, regex, JSON schema, word counts, and semantic similarity scores like ROUGE-L.
    • Model-agnostic execution runs tests via shell commands or HTTP endpoints to support any local or cloud provider.
    • Regression reporting generates Markdown and JSON reports with pass rates aggregated by custom tags.
    • Comparative analysis compares two different runs or prompts side-by-side using McNemar significance testing.

    How it works

    1. Define cases in a JSONL file specifying inputs, expected outputs, and specific check types like contains, json, or refusal.
    2. Execute runs by pointing the harness at your model interface using the run command.
    3. Score results against your dataset to calculate pass rates and latency metrics.
    4. Compare outputs to identify specific wins, losses, and statistical significance between model versions.

    Frameworks & tools

    Built for Python environments. Integrates with any model interface via CLI or HTTP. Uses JSONL for datasets and outputs Markdown for CI/CD reporting.

    Why this beats prompting it yourself

    Manual inspection of LLM outputs doesn't scale and misses subtle regressions in JSON formatting or word counts. This skill automates the scoring logic and statistical comparisons, providing a binary pass/fail signal for CI gates that simple prompting cannot replicate reliably.

    Use cases

    • Prompt engineering to verify if a new system prompt improves accuracy without breaking formatting.
    • Model migration to test if switching from GPT-4 to a local Llama-3 model maintains quality standards.
    • CI/CD integration to block deployments if the pass rate on a golden dataset drops below a defined threshold.

    Known limitations

    Requires users to provide their own execution shim for models. The harness does not include built-in LLM-as-a-judge capabilities.

    Requires a user-provided shell command or HTTP endpoint to interface with models. Does not include internal LLM-as-a-judge scoring logic.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    Permissions

    Terminal / Shell
    Listed1 month ago

    What's inside

    Frequently Asked Questions

    This skill provides a structured framework to measure LLM performance using repeatable test suites, allowing you to catch formatting errors or quality regressions that manual testing often misses.

    The harness is model-agnostic and works with any AI agent or LLM that can be accessed via shell commands or HTTP endpoints, including local models like Llama-3 and cloud APIs like GPT-4.

    You will receive the core evaluation engine, a set of standard validation checks (like JSON schema and ROUGE-L), and templates for generating regression and comparative reports.

    Setup involves defining your test cases in a JSONL file and configuring a simple execution shim to connect the harness to your specific model interface.

    While this harness excels at objective checks like regex, JSON validation, and semantic similarity, it does not currently include built-in LLM-as-a-judge (using one LLM to grade another) capabilities.

    Yes, the harness generates Markdown and JSON reports designed to be consumed by CI/CD pipelines, allowing you to automatically block deployments if pass rates fall below your threshold.

    New in LLM Ops