More screenshots

    Works with the AI tools you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIGitHub CopilotGitHub CopilotGemini CLIGemini CLI+20 more

    LLM Eval Run Comparability & Reproducibility Receipt

    1

    Compare two local LLM evaluation manifests and issue a deterministic receipt showing whether their reported scores are structurally comparable.

    $9

    Secure checkout via Stripe

    30-day refund guarantee

    Converts to your local currency at checkout

    0 installsSecurity scanned
    LLM Eval Run Comparability & Reproducibility Receipt

    LLM Eval Run Comparability & Reproducibility Receipt

    Example session with this skill installed

    Compare baseline.json with candidate.json under policy.json and write a new eval-comparability-packet. Tell me whether the score delta is structurally comparable, and list all controlled and uncontrolled changes.

    • Read your context and instructions
    • Compiled the llm eval run
    • Generated the data export

    LLM Eval Comparability Receipt

    Status: BLOCK
    Run ID: 1eae0fcd9fc17dacc3d0

    The two supplied runs are not yet structurally comparable. One uncontrolled provenance field changed, so the reported score delta should not be interpreted until the BLOCK item is resolved.

    CheckBaselineCandidateControlledDecision
    dataset_sha256aaaa…aaaaffff…ffffNoREVIEW
    prompt_sha256bbbb…bbbbeeee…eeeeYesREVIEW
    repetitions55No changeREADY
    temperature00No changeREADY

    Summary

    • Fields compared: 18
    • Controlled changes: 1
    • Uncontrolled changes: 1
    • Baseline repetitions: 5
    • Candidate repetitions: 5
    • Reported score delta: 0.05

    Issues

    • BLOCK — UNCONTROLLED_CHANGE_LIMIT_EXCEEDED: the policy does not allow this uncontrolled dataset change.
    • REVIEW — CONTROLLED_FIELD_CHANGED: the prompt hash changed and was declared controlled.
    • REVIEW — UNCONTROLLED_FIELD_CHANGED: the dataset hash changed without control.

    Packet written

    1. result.json
    2. comparability-matrix.csv
    3. provenance-delta.csv
    4. issues.jsonl
    5. EVAL_COMPARABILITY.md
    6. evidence-receipt.json

    Resolve every BLOCK item and document each controlled change before interpreting the score difference. This receipt compares supplied provenance; it does not rerun an evaluation, establish statistical significance or decide which system is better.

    llm-eval-run-comparability-reproducibili.csv

    CSV · data export

    Generated

    Example file from a real run - the skill writes it into your workspace.

    Connects securely to your tools. The creator never sees your data.

    What you get

    Compare two local LLM or RAG evaluation manifests before accepting a score delta.Detect dataset, model, prompt, metric and run-setting differences.Produce a reproducibility evidence packet for review or client handoff.Separate structural comparability from statistical or quality judgment.

    About this skill

    The problem

    Two LLM or RAG evaluation runs can show different scores even when their datasets, prompts, model settings or evidence were not held constant. Without a structural comparison, teams may treat an uncontrolled change as a real improvement.

    What it does

    This downloadable local package compares two buyer-supplied evaluation manifests under a policy and writes a deterministic review packet. It includes an installable Python CLI, a local stdio MCP adapter and a companion Agent Skill.

    • Compares 18 provenance and configuration fields.
    • Separates controlled changes from uncontrolled changes.
    • Produces result.json, comparability-matrix.csv, provenance-delta.csv, issues.jsonl, EVAL_COMPARABILITY.md and evidence-receipt.json.
    • Rejects malformed inputs and unsafe paths, never overwrites an existing packet and binds inputs and outputs with SHA-256.

    Best for

    • AI product teams and QA consultants reviewing LLM or RAG experiment runs.
    • Agencies preparing a reproducible client handoff.
    • Teams that need evidence before accepting a reported score change.

    Privacy and requirements

    Processing stays local. No API key, login, upload, telemetry or live external action is included. Requires Windows 11 and Python 3.11–3.14. The local MCP adapter optionally uses mcp==2.2.0.

    Known limitations

    The tool does not execute models, design benchmarks, recompute metrics, claim statistical significance, judge result quality, log into vendors or reproduce an experiment. It evaluates structural comparability only.

    How to install

    Works the same in every agent - Claude, Cursor, Codex, Copilot and 20+ more.

    ~30 seconds
    1. 1

      Download the ZIP

      Free skills download straight away. Paid skills unlock right after purchase.

    2. 2

      Unzip into your skills folder

      Every agent reads skills from one folder on your machine. Drop the unzipped folder in there.

    3. 3

      Ask your agent to use it

      Restart the agent if it was already running. It picks the skill up automatically - no config needed.

    Skills folder by agent

    Click the path to copy it. Create the folder if it does not exist yet.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill passes our trust checks below.

    Security scanned

    Passed our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    30-day refund

    Not a fit? Get your money back

    Trust & safety

    Security scanned

    Verified clean 19 days ago

    • Passed all security checks, Safe to install

    Listed19 days ago
    Updated12 days ago

    What's inside

    Frequently Asked Questions