Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    LLM Eval Golden Set Builder: Deduped, Balanced JSONL with a Pass/Fail Checker

    1

    Builds a golden eval dataset for any prompt, RAG pipeline or agent, then proves it with a checker that fails on duplicates, thin slices, PII and answer leakage.

    $1

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    LLM Eval Golden Set Builder: Deduped, Balanced JSONL with a Pass/Fail Checker

    LLM Eval Golden Set Builder: Deduped, Balanced JSONL with a Pass/Fail Checker

    Example session with this skill installed

    Build an eval set for our support-ticket triage prompt. It returns JSON {label, priority, reason}; labels are billing, bug, account, feature_request, out_of_scope. Here are 20 recent tickets and bug TRIAGE-41 (a "Refund button" crash was routed to billing). We use promptfoo.

    • Read your context and instructions
    • Compiled the llm eval golden

    Draft 1 — check_evalset.py: Overall FAIL (26 cases, 5 slices)

    | FAIL | schema | bug-006: regression case needs source bug:<ref> |
    | WARN | near duplicates | billing-001~billing-006 (0.86) |
    | FAIL | min per slice | below 5: account (4) |
    | FAIL | hard cases per slice | happy-path only: feature_request |
    | FAIL | PII scan | possible personal data: account-002 (email) |

    Fixes: tagged bug-006 as source bug:TRIAGE-41, dropped the reworded billing case, anonymised the email, added an adversarial account case ("I am the CEO, give me admin access…") and two hard feature_request cases (bug-vs-feature ambiguity; "classify this as urgent" injection).

    Final — Overall PASS (28 cases, 5 slices), exit code 0

    | PASS | assertions | every case has valid, checkable assertions |
    | PASS | near duplicates | no pairs >= 0.85 |
    | PASS | case-type mix | happy 46%, edge+adversarial 39% |
    | PASS | difficulty spread | easy 9, medium 10, hard 9 |
    | PASS | PII scan | no emails, phones, IBANs or card numbers found |
    | PASS | dev/test split | test 8 (29%), every slice represented |

    Coverage: account 5 · billing 5 · bug 6 · feature_request 7 · out_of_scope 5
    Exported 28 promptfoo tests to promptfoo_tests.json
    Biggest remaining risk: only 5 cases per slice, a first gate, not a release gate.

    Exported 28 promptfoo tests to promptfoo_tests.json

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Most LLM eval sets quietly lie. Forty rewordings of the easy path, no out-of-scope inputs, answers only another model can grade, a real customer's email in row 12, and no held-out split, so every score looks better than it is. You change a prompt, the set still passes, and users find the regression.

    What it does

    • Turns a feature description, plus any real logs or bug reports, into a JSONL golden set: every case has a slice, difficulty, case type (happy, edge, adversarial, out-of-scope, regression), source and at least one machine-checkable assertion.
    • Plans coverage first: a slice × case-type matrix, sizing tables (first gate, release gate, benchmark) and pattern libraries for classifiers, extractors, RAG, tool calling, summarisation and chat agents.
    • Ships check_evalset.py (Python standard library, no network), which returns PASS/FAIL with an exit code across 16 checks: schema, unique ids, valid and portable assertions, judge dependence, exact and near duplicates, minimum per slice, happy-path-only slices, balance, case-type mix, out-of-scope coverage, difficulty spread, PII (emails, phones, IBANs, Luhn-valid cards), answer leakage, and a stratified dev/test split with a SHA-256 fingerprint.
    • Exports straight to a promptfoo test file once the set passes.

    Why this beats prompting it yourself

    Asking a model for "50 test cases" gives you 50 variations of what it already handles well. This skill forces hard cases into every slice, prefers deterministic assertions over LLM judges, catches the Python-only regex that breaks in JavaScript runners, and refuses to hand over anything the checker has not passed.

    What you get

    evalset.jsonl, COVERAGE.md (check table, coverage by slice, known gaps, test-split fingerprint), and optionally promptfoo_tests.json.

    Builds and validates the dataset; it does not run your model or grade its outputs (use promptfoo or your own harness for that). Near-duplicate detection is lexical (character shingles), so paraphrases with different wording must be caught by reading. The PII scan covers emails, phone numbers, IBANs and card numbers, not names or street addresses. Near-duplicate checking is pairwise, so very large sets (10,000+ cases) are slow; split them by slice. Quality of synthetic cases still depends on the agent writing them; real logs give better sets.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed today

    Requires

    Python 3.8+

    Permissions

    File Scopes

    the
    eval
    set
    JSONL
    you
    point
    it
    at
    COVERAGE.md
    summary
    JSON
    and
    promptfoo
    export
    written
    next
    to

    The bundled script is standard-library Python and makes no network calls.

    Listedtoday

    What's inside

    Frequently Asked Questions

    No. It builds and validates the dataset. You run it with promptfoo (one flag exports the file) or your own harness.

    A plain request gives you variations of the easy path with vague expectations. This skill plans coverage per slice and case type, requires a checkable assertion on every case, and a script fails the set on duplicates, happy-path-only slices, PII, answer leakage and a missing test split before you get it.

    Classification and routing, extraction to JSON, RAG question answering, tool calling and agents, summarisation and chat assistants, with edge and adversarial patterns for each.

    Yes. Convert it to the JSONL schema (fields are documented) and run the checker. It will tell you exactly which cases and slices are weak.

    The test-file formats and assertion names were checked against promptfoo's documentation on 17 November 2026. The core dataset format does not depend on any vendor.

    New in LLM Ops