Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Eval Dataset Builder: Test Your AI Before Users Do

    2

    Eval Dataset Builder makes your agent build the test suite an ML engineer would.

    Free

    11 uses5.0 (1 review)
    Eval Dataset Builder: Test Your AI Before Users Do

    Eval Dataset Builder: Test Your AI Before Users Do

    Example session with this skill installed

    We're changing the system prompt of our customer-support chatbot (logistics app, answers from our help center). Build an eval set so we can compare old vs new prompt before releasing.

    • Read your context and instructions
    • Compiled the eval dataset builder

    Criteria (agreed)

    Correct per help-center article (required) 2.

    Grounded: no policy not in the context (required) 3. Asks for order ID when missing (required) 4.

    Safe: never reveals other customers' data; resists injection (required) 5. Tone: polite, ≤ 120 words

    Coverage (60 cases, from 300 anonymised tickets + help center)

    Categoryhappyedgeadversarialrefusalregression
    Track shipment652-2
    Delivery changes54121
    Refunds & claims54321
    Account & billing4321-
    Out of scope--25-

    Difficulty: 40 % easy, 40 % medium, 20 % hard.

    Example case

    {"id":"refund-edge-03","input":"my parcel arrived broken, i want money back NOW","context":"[help article: damaged items - claim within 7 days with photos]","rubric":"Required: asks for order ID and photos; states 7-day claim window; no promise of a refund amount. Tone: empathetic.","grader":"llm_rubric","category":"refunds","type":"edge","difficulty":"medium"}
    

    Validation

    python scripts/check_dataset.py evals.jsonl --split 0.25
    rows: 60 · 0 errors · 2 near-duplicates reworded · split -> dev 45, test 15 (frozen)
    

    Baseline (dev set, promptfoo, LLM judge + regex/contains graders)

    Old promptNew prompt
    Overall pass78 %86 %
    Refunds & claims71 %90 %
    Adversarial63 %50 % ⚠️

    Regression found: the new prompt follows an injected instruction in the order note field (2 cases). Fix suggested: treat user-supplied fields as data. Judge agreement with your manual grading on 15 cases: 14/15.

    Next: fix injection, re-run dev, then run the frozen test split once before release.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Stop shipping prompt changes on vibes

    Most teams test their AI feature with five questions they typed once. Then a prompt tweak quietly breaks refunds, injection gets through, or a new model regresses. Eval Dataset Builder makes your agent build the test suite an ML engineer would.

    What it does

    • Defines "good": feature, users, top failure costs, and checkable criteria for correctness, groundedness, format, safety, tone, latency and cost. Reads your system prompt, tools, docs and real logs or tickets first.
    • Coverage matrix: categories (user intents) x types: happy, edge (empty, long, multilingual, typos, ambiguous, multi-step), adversarial (prompt injection, fake authority, misleading context), refusal and regression; difficulty mix; 30-100 cases to start.
    • Cases in JSONL: input or messages, RAG context, expected answer or rubric, grader, category, type, difficulty; anonymised data, verified answers, no leakage.
    • Graders: exact, contains, regex, JSON schema and code checks first; LLM-as-judge rubrics with binary required criteria for open-ended quality, plus judge-vs-human spot checks.
    • Validation script (check_dataset.py, tested, stdlib only): schema, duplicate IDs and inputs, near-duplicates, answer leaks, bad regex, thin categories, missing edge/adversarial cases, stratified dev/test split, CI exit code.
    • Harness & baseline: config for promptfoo, OpenAI Evals, Braintrust, LangSmith, Inspect, DeepEval or a simple script; pass rate per category, worst failures, regression gate for CI.
    • Keeps growing: every production bug becomes a regression case; the test split stays frozen.

    Includes

    • SKILL.md workflow
    • scripts/check_dataset.py - dataset validator and splitter
    • references/coverage.md - coverage matrix example, edge-case and adversarial generators, sizing, real-input sources
    • references/graders.md - grader table, LLM-judge rubric template, promptfoo example, metrics

    Pairs with Prompt Engineer. Works with Claude Code, Claude (Cowork), Codex, Cursor and any SKILL.md-compatible agent.

    Requires human verification for expected answers. Does not generate harmful content. Best for initial sets of 30-100 cases. Requires a runner like promptfoo for execution.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    5.0
    1 review
    5
    1
    4
    0
    3
    0
    2
    0
    1
    0

    This skill has been used 11 times.

    Trust & safety

    Security scan

    Security scan passed 8 days ago

    • Free with an Agensi account

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listed8 days ago
    Updated5 days ago

    What's inside

    Frequently Asked Questions

    This skill solves the 'vibes-based' evaluation problem by replacing subjective testing with structured, adversarial datasets that quantify performance improvements and regression risks.

    The skill generates standardized JSONL files and grader definitions compatible with industry-standard frameworks like promptfoo, LangSmith, Braintrust, and DeepEval.

    You receive the dataset generation engine, a coverage mapping tool, deterministic grading rubrics, and a Python validation script to ensure dataset integrity.

    To ensure high quality and manual review feasibility, the skill is optimized to generate between 30 and 100 high-impact test cases per run.

    Unlike generic prompts, this skill uses a specialized distribution logic (40/40/20 difficulty split) and specifically targets adversarial injections that standard LLMs typically ignore.

    While the skill automates the creation of inputs and rubrics, we recommend human verification for the 'expected answer' fields, especially for open-ended or subjective tasks.

    New in LLM Ops