Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Eval Dataset Curator

    1

    Build a gold eval set from production data: stratified, deduplicated, labelled and versioned.

    $15

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Eval Dataset Curator

    Eval Dataset Curator

    Example session with this skill installed

    Our evals are 40 cases we wrote by hand and they all pass while production fails. Help me build a real gold set from 8,000 production runs.

    • Read your context and instructions
    • Compiled the eval dataset curator

    Worked example — illustrative data.

    Eval dataset plan: support agent

    Verdict

    A 240-case stratified set from 14 days of traffic, four-field schema, 86% raw agreement (kappa 0.71), covering 12 intents and 3 difficulty bands. Gaps: voice, non-English, enterprise tier.

    Sampling plan

    StratumPopulationSample
    Order status3,10048
    Refunds1,90048
    Returns policy1,20036
    Address change70024
    Escalation60024
    Other (7 intents)50060

    Stratified by difficulty (short/long/tool-heavy) and outcome (pass/fail), across 14 days.

    Label schema

    • Verdict: pass / fail / partial, with one-line definitions.
    • Failure class: wrong fact, missing action, unsafe, formatting, tone.
    • Difficulty: easy / medium / hard, defined by domain knowledge needed.
    • Confidence: high / low; low goes to adjudication.

    Labelling guide

    Three worked examples per verdict, including the borderline case: a correct answer that failed to confirm a refund amount.

    Deduplication

    Token-overlap above 0.8 removed 61 cases. Kept the version with the richest metadata.

    Leakage checks

    • Answer in input: 4 cases removed.
    • Mock-dependent: 9 flagged, kept with mocks pinned.
    • Time-dependent: 12 excluded (prices change weekly).

    Maintenance

    Additions need a second label; removals need a reason. Version as support-gold-v1, v2.

    What this set cannot tell you

    Nothing about voice, non-English, or enterprise-specific tools.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Eval dataset curation from production traffic: dedupe, stratify, label, and document coverage so the scores mean something.

    Use this skill when:

    • build an eval set
    • measure quality without gold data
    • evals do not reflect production
    • label and stratify test cases

    Who it's for: AI teams whose evals do not match reality.

    Example request

    Build an eval set from six months of support tickets.

    What the example returns

    • 1,200 cases, deduped to 480
    • Stratified by intent and channel
    • Gold labels, agreement 0.91

    In the package: five files — SKILL.md with the trigger and workflow, README.md, references/METHOD.md (the full method), references/CHECKLIST.md (the pass/fail gate) and examples/EXAMPLE.md (a worked run you can copy).

    Works with Claude Code, Cursor, Codex CLI and any agent that reads SKILL.md. Plain markdown: no API key, no account, no network access.

    Eval Dataset Curator

    Does not provide an eval harness or execution engine. Requires human domain experts for ground truth labeling. Small datasets (under 150 cases) may lack statistical significance.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 2 days ago

    Permissions

    Read Files
    Listed3 days ago
    Updated2 days ago

    What's inside

    Frequently Asked Questions

    The skill identifies and extracts 150 to 300 high-signal cases from your raw production logs, organizing them into a stratified 'gold set' that accurately represents your actual user traffic rather than just random edge cases.

    This skill is framework-agnostic and compatible with any AI agent or LLM system that can export execution traces, JSON logs, or CSV files from tools like LangSmith or Arize Phoenix.

    You receive the stratified sampling plan, the custom four-field labeling schema, the deduplicated dataset, and the documented adjudication reasoning for resolving labeling disagreements.

    No, this skill focuses on the data curation and ground-truth establishment phase; you will still need a human domain expert to provide the final verdicts and a separate tool to run the actual benchmark tests.

    By identifying and removing near-identical cases and auditing for data contamination, the skill ensures your 'gold set' doesn't contain redundant data that could artificially inflate your model's performance scores.

    New in LLM Ops